Weaviate

Call Weaviate from Flux tasks for vector search and multi-tenant collections.

Weaviate is an open-source vector database, available self-hosted via Docker or as a managed service through Weaviate Cloud. Flux has no native client — use the weaviate-client v4 SDK from inside @task.

Setup options

Self-hosted (Docker). A single container is enough for development:

# docker-compose.yml
services:
  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.27.0
    ports:
      - "8080:8080"
      - "50051:50051"
    environment:
      AUTHENTICATION_APIKEY_ENABLED: "true"
      AUTHENTICATION_APIKEY_ALLOWED_KEYS: "dev-key-please-rotate"
      AUTHENTICATION_APIKEY_USERS: "admin"
      AUTHORIZATION_ADMINLIST_ENABLED: "true"
      AUTHORIZATION_ADMINLIST_USERS: "admin"

Weaviate Cloud. Sign up, create a cluster, copy the REST endpoint and a tenant API key from the dashboard.

Install

pip install weaviate-client

The v4 client (weaviate-client>=4.0) reworked the API; v3 examples on the wider internet do not apply.

API key

Store the key in Flux’s secret store:

flux secrets set weaviate_api_key wv_...
flux secrets set weaviate_url https://your-cluster.weaviate.network

Connect, write, query

from flux import workflow, task, ExecutionContext


@task.with_options(secret_requests=["weaviate_api_key", "weaviate_url"])
async def add_articles(secrets, articles: list[dict]) -> int:
    import weaviate
    from weaviate.classes.init import Auth

    with weaviate.connect_to_weaviate_cloud(
        cluster_url=secrets["weaviate_url"],
        auth_credentials=Auth.api_key(secrets["weaviate_api_key"]),
    ) as client:
        coll = client.collections.get("Article")
        with coll.batch.dynamic() as batch:
            for a in articles:
                batch.add_object(properties=a)
        return len(articles)


@task.with_options(secret_requests=["weaviate_api_key", "weaviate_url"])
async def search(secrets, query: str, k: int = 5) -> list[dict]:
    import weaviate
    from weaviate.classes.init import Auth
    from weaviate.classes.query import MetadataQuery

    with weaviate.connect_to_weaviate_cloud(
        cluster_url=secrets["weaviate_url"],
        auth_credentials=Auth.api_key(secrets["weaviate_api_key"]),
    ) as client:
        coll = client.collections.get("Article")
        result = coll.query.near_text(
            query=query,
            limit=k,
            return_metadata=MetadataQuery(distance=True),
        )
        return [
            {"props": o.properties, "distance": o.metadata.distance}
            for o in result.objects
        ]

connect_to_weaviate_cloud is the helper for managed clusters. For local Docker use connect_to_local(host="localhost", port=8080). The v4 client manages a long-lived gRPC connection and needs the with block — leaking a client across @task boundaries leaks the connection.

Multi-tenant collections

Weaviate’s multi-tenancy is first-class and fits Flux’s namespace model directly. Enable it on the collection:

from weaviate.classes.config import Configure
client.collections.create(
    name="Article",
    multi_tenancy_config=Configure.multi_tenancy(enabled=True),
)

Then attach a tenant before reading or writing:

coll = client.collections.get("Article").with_tenant("acme")

A natural mapping is tenant = workflow_namespace. Build it into a helper task so every read and write is scoped automatically — forgetting .with_tenant(...) on a multi-tenant collection raises immediately, which is the right failure mode.

What goes wrong

v3 → v4 client confusion. Most Stack Overflow answers and old blog posts are still on v3. client.batch.configure(...), client.data_object.create(...), etc. are all v3 — they don’t exist in v4.
Auth at the wrong layer. Self-hosted Weaviate ships open by default. Setting AUTHENTICATION_APIKEY_ENABLED without AUTHORIZATION_ADMINLIST_ENABLED accepts the key but rejects every operation as unauthorized.
Cluster URL must include the scheme. connect_to_weaviate_cloud(cluster_url="your-cluster.weaviate.network") fails. It must be https://your-cluster.weaviate.network.