Load data from BigQuery into ClickHouse Cloud in a few clicks for fast exploration and prototyping. The connector simplifies data migration for testing ClickHouse's real-time query performance on your BigQuery datasets.


Summary

Running ClickHouse Cloud as a _speed-layer_ on top of BigQuery is an increasingly common pattern: you let BigQuery handle batch processing and data warehousing, while ClickHouse powers **interactive queries that need sub-second latency**. As we onboard more customers with this architecture, it became clear that moving data over is a significant first hurdle in the onboarding experience. Teams would spend days iterating on export scripts, writing custom type conversion logic, and trial-and-error — before they could even start prototyping and testing their queries.

> [*Join the waitlist*](https://clickhouse.com/cloud/clickpipes/bigquery-connector) *today to try out the BigQuery connector for ClickPipes!*

Today, we’re announcing the first step towards making this integration a no-brainer: **a new BigQuery connector for ClickPipes**, in Private Preview! Whether you're evaluating ClickHouse Cloud for specific use cases where sub-second query latency matters, or exploring how BigQuery and ClickHouse can work together in your data architecture, the connector provides a straightforward path to get your data into ClickHouse Cloud for rapid prototyping and performance testing.

<iframe width="768" height="432" src="https://www.youtube.com/embed/9skyBoIQbTM?si=y3LwPrrj7s5Vsi6N" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

## Why build a ClickPipe?

**Simpler than building your own pipeline.** Previously, loading data from BigQuery into ClickHouse meant manually exporting tables to GCS, managing service account permissions across both platforms, writing custom SQL, and handling schema mapping yourself. The BigQuery ClickPipe handles all of this automatically: point it at your BigQuery dataset(s), configure your target tables, and the connector manages all those steps for you.

**Built for continuous ingestion.** The connector is built on [PeerDB](http://github.com/PeerDB-io/peerdb), an ✨open-source✨ data replication platform maintained by ClickHouse. Although BigQuery doesn't have a replication log for log-based CDC, PeerDB has the primitives to support query-based CDC for incremental syncs. We'll use this to add continuous ingestion capabilities in future releases as we evaluate the right cost-performance tradeoffs for BigQuery’s architecture.

**Data movement is just the start.** Loading your BigQuery tables into ClickHouse gets you far: you can prototype queries, validate performance, and build proof-of-concepts for the interactive workloads you want to migrate. But a successful, end-to-end workload migration also means translating SQL queries, optimizing schemas for ClickHouse's architecture, and validating results across (potentially thousands of) queries. As our teams make progress on AI-assisted migrations (see [clickhouse.build](https://clickhouse.com/blog/clickhouse-build-agentic-cli-accelerate-postgres-clickhouse-apps) for a recent example), we're exploring how to package this up for BigQuery and other connectors, too!

## Main features

#### Efficient bulk loads

Private Preview supports **bulk data loads** from BigQuery tables. This is what most teams need as that first step: load a representative dataset, test some queries, and validate that ClickHouse performs the way you expect. The connector exports each table as compressed Avro files (auto-sharded at 1GB per file) to an intermediate GCS bucket, then loads them into ClickHouse Cloud in parallel. For very large datasets, it’s possible to adjust ingestion settings and sizing to speed up the initial load.

#### Table- and column-level filtering

The connector automatically discovers all datasets and tables within your BigQuery project (that the associated service account is scoped to). From the ClickPipes UI, you have fine-grained control over what to load, whether that's a single table, an entire dataset, or even tables across different datasets.

![image2.png](/uploads/image2_4fd922a530.png)

For each table, you can exclude specific columns, customize column mappings, and configure granular settings like how to handle `NULL` values. This eliminates the manual work of exporting tables one-by-one and gives you control over what data lands in ClickHouse without writing any code.

#### Implicit type mapping

The connector handles BigQuery's type system automatically, including complex conversions that don't map directly to ClickHouse (*e.g.* `DATE` values stored as days-since-epoch become proper ClickHouse dates, `BYTES` columns get base64 decoded). With type mapping out of the way, you avoid writing error-prone conversion logic and investing time in debugging edge cases when data doesn't load correctly or queries return unexpected results.

#### Terraform and Open API support

Although you can only sync full tables in Private Preview, for smaller tables it can be useful to do full periodic reloads. To support workflow automation and managing the ClickPipe as code, a new `bigquery` source type is available in [the ClickPipes OpenAPI specification](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/ClickPipes), as well as the ClickHouse Terraform Provider ([3.8.3-alpha1](https://github.com/ClickHouse/terraform-provider-clickhouse/releases/tag/v3.8.3-alpha1)+).

## How does it work?

Building a connector for a cloud data warehouse presents a unique challenge: moving (potentially very) large datasets efficiently while minimizing upstream costs. The new ClickPipes connector uses BigQuery's native export jobs (**free!**) to avoid incurring query processing charges.

![image1.jpg](/uploads/image1_a424ad43f3.jpg)

***Bigquery ClickPipe:** the user configures a ClickPipe to point to the desired tables in BigQuery, providing a service account key for authentication (1). A BigQuery job exports the data into an intermediate GCS bucket (2-3). ClickPipes reads from the staging bucket (4) and inserts data into ClickHouse Cloud tables (5). Authentication between services is managed via IAM.*

**Setup requirements:**

* A BigQuery project with the dataset(s) you want to export tables from, and a [ClickHouse Cloud account](https://clickhouse.com/cloud).
* A Google Cloud service account with BigQuery `dataViewer` and `jobUser` permissions, as well as GCS `objectAdmin` and bucketViewer permissions.  
* A dedicated GCS bucket for intermediate staging. In Private Preview, this bucket must be provisioned in your GCP account. In the future, ClickPipes will provision and manage the staging bucket.

## Next up: incremental syncs

<p>
After a successful onboarding, the most common request we get from customers is performing incremental loads to keep BigQuery and ClickHouse in sync. Unlike databases with transaction logs, BigQuery requires querying tables to detect changes, or using advanced ($$) features like time travel — all of which drive up costs. &#128200; We're evaluating strategies for query-based CDC like timestamp-based incremental loads for append-only tables, partition-aware sync for time-partitioned data, and metadata-driven triggers to minimize query costs. <strong>If incremental syncs are critical for your use case, we’d love to work with you as a design partner</strong>.
</p>

In the meantime, we recommend [using the Google Cloud Storage ClickPipe](https://clickhouse.com/docs/integrations/clickpipes/object-storage/gcs/overview) to continuously sync BigQuery data exports to ClickHouse Cloud, once the initial load via the BigQuery ClickPipes is completed. We’ll publish guides on how to implement this pattern in the next few weeks!

## How to sign up for Private Preview?

The BigQuery connector is in Private Preview and available upon request. Fill out [this form](https://clickhouse.com/cloud/clickpipes/bigquery-connector) to join the waitlist, or reach out to your ClickHouse Cloud account manager! We'll work through requests as soon as possible, and expect to reach out to you, worst case, within a few days of signing up.

We're actively looking for design partners to help shape this connector, in particular teams who need incremental syncs, or are planning large-scale BigQuery migrations. Your requirements and feedback will directly influence what we build next!


In our benchmarks, ClickHouse delivers up to 4x faster queries with significant cost savings for interactive analytics. 

Ready to test ClickHouse with your BigQuery data?

Load data from BigQuery into ClickHouse Cloud in a few clicks for fast exploration and prototyping. The connector simplifies data migration for testing ClickHouse's real-time query performance on your BigQuery datasets.

large_image3.jpg

medium_image3.jpg

small_image3.jpg

thumbnail_image3.jpg

image3.jpg

BigQuery connector for ClickPipes is now in Private Preview

Replicate data from MongoDB into ClickHouse Cloud in just a few clicks for blazing-fast analytics on document-based data. Now with more connectivity options, improved reliability, and support for DocumentDB.

<p>The <a href="https://clickhouse.com/cloud/clickpipes">ClickPipes MongoDB CDC connector</a> is now in Public Beta! &#128640; </p>

After working with dozens of early access customers during Private Preview, we've added support for **sharded clusters** and **Amazon DocumentDB**, improved **reliability** for production workloads, and made it easier to **securely connect** to private MongoDB deployments.

Whether you're running a one-time migration or continuously replicating operational data for analytics, the connector delivers up to **100x faster queries** without the overhead of managing custom pipelines.

> [Sign up for ClickHouse Cloud](https://console.clickhouse.cloud/signup) today to try out the [MongoDB CDC connector for ClickPipes](https://clickhouse.com/cloud/clickpipes)!

<iframe width="768" height="432" src="https://www.youtube.com/embed/ZWM6xsaF23M?si=TlAAhBb7J5aav-pI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>

Here’s what our customers are saying:

>Before using ClickPipes, we relied on multiple tools to replicate MongoDB data into ClickHouse, many of which required constant maintenance or manual intervention. After switching to the MongoDB connector in ClickPipes, those issues disappeared: the initial setup was simple, and once configured, it required no ongoing attention. Using materialized views makes data modeling flexible and easy to evolve.
> 
> <br />
>Overall, ClickPipes has significantly reduced operational overhead and has worked great to replicate several terabytes of business data a day for analytical workloads. - <a href="https://www.rapidata.ai">Rapidata</a>

## What’s new in Public Beta?

### Broader platform compatibility

The MongoDB CDC connector initially launched with support for single [replica sets](https://www.mongodb.com/docs/manual/core/replica-set-members/). We’ve now extended support to [**sharded clusters**](https://www.mongodb.com/docs/manual/core/sharded-cluster-components/), enabling replication from common MongoDB topologies at any scale — whether you're running a single replica set or a distributed deployment with data spread across multiple shards.

We've also added compatibility with **Amazon DocumentDB**, a managed document database service that is API-compatible with MongoDB. See our [new integration guide](https://clickhouse.com/docs/integrations/clickpipes/mongodb/source/documentdb) for step-by-step instructions on how to configure DocumentDB for change stream replication.

### Secure connectivity options

To allow secure connections to MongoDB instances in private networks, which is a table stakes requirement for production environments, you can now configure both **AWS PrivateLink** and **SSH tunneling** when creating a new MongoDB ClickPipe. We've also added support for [X.509 certificate authentication](https://www.mongodb.com/docs/manual/core/security-x.509/), enabling **mutual TLS authentication** between the connector and your MongoDB deployment.

### Production-readiness

We've made significant improvements to reliability and observability based on Private Preview feedback. The connector now handles <strong>data type edge cases</strong> more gracefully (*e.g.* very large floats, dates outside the standard range) and uses enhanced error classification to <strong>automatically retry transient failures</strong> that would previously trigger unnecessary alerts.

<p>We've also refined the initial snapshot phase with smarter batching logic that measures uncompressed data size to <strong>prevent out-of-memory issues</strong>. This addresses the most significant operational issue we identified with large-scale deployments during early access: MongoDB's high JSON compression ratios caused batch sizes to balloon unexpectedly when decompressed during ingestion of very large collections. &#128165;</p>

## Main features

In addition to these Public Beta improvements, the MongoDB CDC connector provides several core capabilities designed to make real-time replication from MongoDB to ClickHouse Cloud reliable and performant:

### Real-time replication

The connector leverages MongoDB's native [Change Streams](https://www.mongodb.com/docs/manual/changestreams/) interface to capture all document changes with low latency, ensuring near real-time synchronization between MongoDB and ClickHouse Cloud. This combination creates a powerful architecture for modern applications that need both operational flexibility and analytical performance, with up to 100x faster analytics — we’re calling it the *Document Data Stack*.

![image1.png](/uploads/image1_cba4cd460a.png)

Keep your MongoDB clusters focused on serving mission-critical operational workloads, including high-throughput CRUD operations and efficient document lookups, while ClickHouse Cloud handles complex analytical queries, reports, and business intelligence workloads without impacting upstream performance.

### Advanced JSON Support

To preserve MongoDB’s rich document structures and provide high-precision replication, the connector uses ClickHouse's powerful [native JSON data type](https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse). This enables **high-performance** analytical queries on semi-structured data, at the same time o**ptimizing compression and reducing storage cost**.

### Fully managed experience

The connector is fully integrated into the ClickHouse Cloud experience. It provides **built-in metrics and monitoring** for visibility into replication health, **detailed logs** for error diagnosis and debugging, **in-place pipe editing**, and more.

## What’s next?

As we work towards General Availability (GA), we're focused on making the connector faster and more flexible for production workloads.

* **Parallel snapshot ingestion:** as is, the connector snapshots collections using a single thread, which can lead to a very long initial snapshotting phase. We’re evaluating (logical) partitioning strategies to improve initial load times for large collections.

* **Flattened mode:** we’ll add an option to automatically map top-level document fields to target columns. This will make it easier to model and query replicated data, while preserving schema evolution capabilities.

* **OpenAPI and Terraform support:** for teams managing infrastructure as code, MongoDB ClickPipes will also be available via Open API and Terraform, similar to other ClickPipe types.

**A note on billing:** usage of MongoDB CDC ClickPipes continues to be **free** until GA. Customers will be notified ahead of the GA launch to review and optimize their ClickPipes usage. You can estimate future costs by referring to [billing for Postgres CDC ClickPipes](https://clickhouse.com/docs/cloud/reference/billing/clickpipes).

## Getting started with the MongoDB CDC connector

The MongoDB CDC connector is available to new and existing ClickHouse Cloud customers, in all service tiers. To get started, navigate to the *Data Sources* tab in the ClickHouse Cloud console, configure the connection details for your MongoDB database, and you’re good to go! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for MongoDB ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/mongodb).


Ready to accelerate analytics on MongoDB data? Try the MongoDB CDC connector today and experience a fully managed, native integration experience with ClickHouse Cloud - the world’s fastest analytics database.

Try the MongoDB CDC connector today

large_MongoDB ClickPipe Banner.jpg

medium_MongoDB ClickPipe Banner.jpg

small_MongoDB ClickPipe Banner.jpg

thumbnail_MongoDB ClickPipe Banner.jpg

MongoDB ClickPipe Banner.jpg

MongoDB CDC connector for ClickPipes is now in Public Beta

Ingest data from Amazon S3 into ClickHouse Cloud in **any** **order** for event-driven, blazing-fast analytics. Files are no longer required to follow lexicographical order.

When you’re on the receiving end of a data integration pipeline, you quickly learn that real-world data knows no rules. Sometimes it arrives, sometimes it doesn’t. Then it arrives twice, or maybe out-of-order. For object storage, in particular, this creates an interesting problem: how do you efficiently track what's new without scanning too many objects all the time?

Like most tools out there, [ClickPipes](https://clickhouse.com/cloud/clickpipes?utm_source=google.com&utm_medium=paid_search&utm_campaign=21862172336_181837693625&utm_content=764403839926&utm_term=clickpipes_g_c&gad_source=1&gad_campaignid=21862172336&gclid=CjwKCAjwisnGBhAXEiwA0zEOR3kM4ZBQO0_NpfRPpYU2YWG5onZqwF-Kf8m845Ol2IJeyN96r0k8DxoCit4QAvD_BwE) didn’t have a good solution for this problem, and simply expected you to ensure there were *some* rules to guarantee files landed in your bucket in lexicographical order. **Not anymore!**

![image6.png](/uploads/image6_b5104fce2a.png)

We’ve added a new mode to the S3 ClickPipe that allows ingesting files in any order using S3 Event Notifications via [Amazon SQS](https://aws.amazon.com/sqs/). Whether you’re dealing with backfills, retries, late-arriving data, or some other source of out-of-orderness — we handle it.

Sign up for ClickHouse Cloud

## Why is this a big deal?

By default, the S3 ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. Besides not rolling off the tongue, lexicographical order means that any new file *must* be lexically greater than the last ingested file. For example, files named `events_2024-12-01.parquet`, `events_2024-12-02.parquet`, and `events_2024-12-03.parquet` will be ingested in order, but if a backfill named `events_2024-11-30.parquet` lands later in the bucket, it will be ignored.

This is not great.

With unordered mode, this limitation no longer applies: instead of polling your bucket every 30 seconds looking for the next file to process, the ClickPipe waits to be notified that a new file has landed. When a notification arrives, ClickPipes just goes and processes that file regardless of its relative order to previously processed files. Although this requires you to configure event notifications, it's a more robust and performant alternative to the otherwise common approach of listing all objects in scope on every sync — which scales poorly and gets expensive fast when you have millions of objects.

## How does it work?

To configure an S3 ClickPipe to ingest files that don’t have an implicit order, you need to set up an [Amazon Simple Queue System (SQS)](https://aws.amazon.com/sqs/) queue connected to the bucket. ClickPipes can then listen for `ObjectCreated` events in the queue that match the specified path.  

![ClickPipes S3 Blog Banner.jpg](/uploads/Click_Pipes_S3_Blog_Banner_08474993cb.jpg)

**Unordered mode:** Files land in S3 in **any order** (A) and trigger SQS notifications (B-C). ClickPipes polls and processes files in the specified path, using a metadata store to track state (1-5). Data is inserted into the target tables with exactly-once guarantees (6).

***“What about failures?”*** Since the steps above span multiple systems and don’t happen in one big transaction™️, failures can occur at any step — reading from S3, marking files as processed, inserting data into ClickHouse, and so on. If a failure occurs, ClickPipes automatically reprocesses the batch. *“**But what about duplicates?”*** Even if files are reprocessed multiple times, the S3 ClickPipe guarantees exactly-once semantics, so no duplicates make it into your target table.

Let’s see it working!

### Create an Amazon SQS queue

*The following instructions assume you already have an S3 bucket with some data in it, as well as enough permissions to manage IAM roles and create new resources in your AWS account.*

**1.** In the AWS Console, navigate to **Simple Queue Service > Create queue**. Use the defaults to create a new queue.

**2.** Edit the queue trust policy to allow your bucket (`<bucket-arn>`) in your AWS account (`<aws-account-id>`) to send messages to the SQS queue (`<queue-arn>`).

```
{
    "Version":"2012-10-17",                   
    "Id": "example-ID",
    "Statement": [
        {
            "Sid": "AllowS3ToSendMessage",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "SQS:SendMessage"
            ],
            "Resource": "<sqs-queue-arn>",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "<bucket-arn>"
                },
                "StringEquals": {
                    "aws:SourceAccount": "<aws-account-id>"
                }
            }
        }
    ]
}
```

**3.** In the ClickHouse Cloud console, navigate to **Settings > Network security information** and copy the IAM role ARN for your service.

![image2.png](/uploads/image2_2accfed654.png)

**4.** Back in the AWS Console, go to **IAM > Roles > Create role**. Choose **Custom trust policy** and paste the IAM role ARN for your ClickHouse Cloud service.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAssumeRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<ch-cloud-arn>"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

**5.** Create an inline policy for the IAM role with the required permissions to scan and fetch objects from the S3 bucket and manage messages in the SQS queue.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3BucketMetadataAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": "<bucket-arn>"
    },
    {
      "Sid": "AllowGetListObjects",
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": "<bucket-arn>/*"
    },
    {
      "Sid": "SQSNotificationsAccess",
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:ListQueues",
        "sqs:ReceiveMessage",
        "sqs:GetQueueAttributes"
      ],
    "Resource": "<sqs-queue-arn>"
    }
  ]
}
```

That was a lot of IAM config, but you’re ready to create a ClickPipe to continuously ingest data from your bucket whenever a new file lands!

### Create a ClickPipe

**1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe**, then choose **Amazon S3**. Enter the details to connect to your S3 bucket, using **IAM role** as the authentication method with the ARN role you created in the previous section.

![image5.png](/uploads/image5_19483129ab.png)

**2.** Click **Incoming data**. Toggle on **Continuous ingestion**, where you’ll see the new **Any order** ingestion option.

![image3.png](/uploads/image3_40f8b8de3b.png)

**3.** Click **Parse information**. Define a **Sorting key** for the target table (very important) and make any necessary adjustments to the mapped schema. Finally, configure a role for the ClickPipes database user.

![image1.gif](/uploads/image1_f958d131c1.gif)

<p>
<strong>4. Sit back and relax.</strong> ClickPipes will now perform an initial scan of your bucket, then start processing files as new notification events arrive. &#128640;
</p>

If that seems like a lot of clicking, the good news is that ClickPipes is fully supported in the [ClickHouse Terraform provider](https://registry.terraform.io/providers/ClickHouse/clickhouse/latest/docs), so you can configure this setup as code from start to finish.

## What’s next?

We’re excited to enable more complex ingestion patterns from object storage into ClickHouse Cloud with the new unordered mode in S3 ClickPipes — you get the same fully-managed, full-speed experience with a little extra flexibility\! For now, this feature is only available for Amazon S3 as a data source. We plan to extend support to Google Cloud Storage (GCS) and Azure Blob Storage (ABS) in the near future, to ensure feature parity across all Object Storage ClickPipes.

If you have any feedback or run into any snags while setting up ClickPipes, reach out to our team\! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for S3 ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/object-storage/s3/overview#continuous-ingestion-any-order).

Ready to eliminate your ETL complexity and reduce your data movement costs? Try the S3 ClickPipe connector today and experience a fully managed, native integration experience with ClickHouse Cloud - the world’s fastest analytics database.

Get started today

Ingest data from Amazon S3 into ClickHouse Cloud in any order for event-driven, blazing-fast analytics.

large_ClickPipes S3 Blog Banner.png

medium_ClickPipes S3 Blog Banner.png

small_ClickPipes S3 Blog Banner.png

thumbnail_ClickPipes S3 Blog Banner.png

ClickPipes S3 Blog Banner.png

Unordered mode for S3 ClickPipes is now available

![azure-blog-banner.png](/uploads/azure_blog_banner_5fda937064.png)

We're excited to announce that the **Azure Blob Storage connector for [ClickPipes](https://clickhouse.com/cloud/clickpipes?utm_source=google.com&utm_medium=paid_search&utm_campaign=21862172336_181837693625&utm_content=764403839926&utm_term=clickpipes_g_c&gad_source=1&gad_campaignid=21862172336&gclid=CjwKCAjwisnGBhAXEiwA0zEOR3kM4ZBQO0_NpfRPpYU2YWG5onZqwF-Kf8m845Ol2IJeyN96r0k8DxoCit4QAvD_BwE)**  is now Generally Available! You can seamlessly load files from Azure Blob Storage into ClickHouse Cloud with even **better performance**, and leverage the new support for **programmatic access** (via OpenAPI and Terraform) to manage your ClickPipes with confidence.

Over the past few months, the connector has gone through an extensive beta phase, with customers moving anywhere between a few GiB to **as much as 60+ PiB of data** into ClickHouse Cloud — validating its reliability and scalability for both small- and enterprise-scale data lake analytics workloads.

> [Sign up for ClickHouse Cloud](https://console.clickhouse.cloud/signup) today to try out the [Azure Blob Storage connector for ClickPipes](https://clickhouse.com/cloud/clickpipes)!

![azure-blog.gif](/uploads/azure_blog_d52244e13f.gif)

## What's new?

Based on customer feedback, and to align the connector with the product experience for other object storage ClickPipes, the Azure Blob Storage connector has been upgraded with:

### Custom HTTP client for better performance

From the rollout of ClickHouse v25.8 to ClickHouse Cloud, the connector will leverage a new, custom HTTP client that delivers more consistent and better performance than the original Azure SDK native client. This brings the connector to parity with object storage ClickPipes for other cloud providers, when it comes to performance.

### Terraform and OpenAPI support

For mature ClickPipes deployments using workflow automation and Infrastructure as Code (IaC), a new `azureblobstorage`  source type is available in [the ClickPipes OpenAPI specification](https://clickhouse.com/docs/cloud/manage/api/swagger#tag/ClickPipes), as well as the ClickHouse Terraform Provider ([3.6.0-alpha1](https://github.com/ClickHouse/terraform-provider-clickhouse/releases/tag/v3.6.0-alpha1)+). Here's an example of how you'd configure an Azure Blob Storage ClickPipe resource in Terraform:

> **Note**: ClickPipes resources are only supported in [alpha releases](https://registry.terraform.io/providers/ClickHouse/clickhouse/3.7.0-alpha1/docs/resources/clickpipe) of the ClickHouse Terraform provider. Regardless, support for object storage ClickPipes is considered **stable**.

```
resource "clickhouse_clickpipe" "azure_blob" {
  name        = "Azure Blob Storage ClickPipe"
  service_id = var.service_id

  source = {
    object_storage = {
      type    = "azureblobstorage"
      format  = "JSONEachRow"

      path                  = var.azure_path              
      azure_container_name  = var.azure_container_name
      connection_string     = var.azure_connection_string

      authentication = "CONNECTION_STRING"
    }
  }

  destination = {
    table         = "my_azure_data_table"
    managed_table = true

    table_definition = {
      engine = {
        type = "MergeTree"
      }
    }

    columns = [
      {
        name = "id"
        type = "UInt64"
      }, {
        name = "name"
        type = "String"
      }, {
        name = "timestamp"
        type = "DateTime"
      }
    ]
  }

  field_mappings = [
    {
      source_field      = "user_id"
      destination_field = "id"
    }, {
      source_field      = "user_name"
      destination_field = "name"
    }, {
      source_field      = "created_at"
      destination_field = "timestamp"
    }
  ]
}
```

In addition to these improvements, it's worth recapping the base capabilities of the Azure Blob Storage connector:

### Built for reliability

Built on top of the [azureBlobStorage](https://clickhouse.com/docs/sql-reference/table-functions/azureBlobStorage) table function, with additional layers of reliability to handle real-world scale. The connector gracefully handles **automatic retries** on failures and guarantees **exactly-once delivery semantics**, so you don't have to worry about data consistency or partial loads.

### Secure by design

Supports ingestion from **private buckets** using connection strings with account credentials or storage account URLs. You can also use **Shared Access Signatures (SAS)** within the connection string for secure, time-limited access.

### Optimized for performance

The connector dynamically adjusts **ingestion parallelism** and **ClickHouse tuning** based on your cloud instance size and workload. It's built to move terabytes of data quickly and efficiently with no need for manual tuning.

### Fully managed experience

The connector is fully integrated into the ClickHouse Cloud experience. It provides **built-in metrics and monitoring, detailed logs** for error diagnosis and debugging, **in-place pipe editing** (e.g., adding columns), and more.

## Pricing

Object storage ClickPipes have a predictable pricing model based on data volume, with minimal added compute cost. The Azure Blob Storage connector for ClickPipes is **10-25x more cost-effective** than using third-party ETL tools, reflecting our commitment to offering ClickPipes as a seamless but affordable option to connect ClickHouse Cloud and your various data sources. For more details, see the [ClickPipes billing documentation](https://clickhouse.com/docs/cloud/reference/billing/clickpipes/streaming-and-object-storage).

## Getting started with the Azure Blob Storage connector

The Azure Blob Storage connector is available to new and existing ClickHouse Cloud customers, in all service tiers. To get started, navigate to the Data Sources tab in the ClickHouse Cloud console, configure the connection details for your storage container, and you're good to go! For step-by-step instructions, frequently asked questions, and gotchas, check out the [documentation for object storage ClickPipes](https://clickhouse.com/docs/integrations/clickpipes/object-storage).

> Ready to eliminate your ETL complexity and reduce your data movement costs? [Try the Azure Blob Storage connector today](https://clickhouse.com/cloud) and experience a fully managed, native integration experience with ClickHouse Cloud — the world's fastest analytics database.



We’re excited to announce the general availability of the Azure Blob Storage connector for ClickPipes. Natively integrated with ClickHouse Cloud, this connector enables blazing-fast load from Azure Blob Storage with just a few clicks.

Marta Paes

Blog posts by Marta Paes

BigQuery connector for ClickPipes is now in Private Preview

MongoDB CDC connector for ClickPipes is now in Public Beta

Unordered mode for S3 ClickPipes is now available

Azure Blob Storage connector for ClickPipes is now Generally Available