Weaviate Vector Database Guide: AI-Native Search for Australian Businesses

Weaviate distinguishes itself through AI-native design decisions that simplify building intelligent applications. Understanding these differentiators helps you evaluate whether Weaviate fits your needs.

Core Differentiators

Weaviate vs Other Vector Databases

Weaviate offers multiple deployment options from local Docker instances to managed cloud. Let's start with local development and understand the core concepts.

Local Development with Docker

# docker-compose.yml for local Weaviate with OpenAI vectorisation
version: '3.4'
services:
  weaviate:
    image: semitechnologies/weaviate:latest
    ports:
      - "8080:8080"
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
      ENABLE_MODULES: 'text2vec-openai,generative-openai'
      OPENAI_APIKEY: 'your-openai-api-key'
      CLUSTER_HOSTNAME: 'node1'
    volumes:
      - weaviate_data:/var/lib/weaviate

volumes:
  weaviate_data:

# Start Weaviate
docker-compose up -d

# Verify it's running
curl http://localhost:8080/v1/meta

Python Client Setup

# pip install weaviate-client

import weaviate
from weaviate.classes.init import Auth

# Connect to local instance
client = weaviate.connect_to_local()

# Or connect to Weaviate Cloud
client = weaviate.connect_to_weaviate_cloud(
    cluster_url="your-cluster-url.weaviate.network",
    auth_credentials=Auth.api_key("your-weaviate-api-key"),
    headers={
        "X-OpenAI-Api-Key": "your-openai-api-key"
    }
)

# Verify connection
print(client.is_ready())

Creating Your First Schema

# Define a schema for Australian business documents
from weaviate.classes.config import Configure, Property, DataType

client.collections.create(
    name="BusinessDocument",
    description="Australian business documents for semantic search",
    vectorizer_config=Configure.Vectorizer.text2vec_openai(
        model="text-embedding-3-small"
    ),
    generative_config=Configure.Generative.openai(
        model="gpt-4"
    ),
    properties=[
        Property(name="title", data_type=DataType.TEXT),
        Property(name="content", data_type=DataType.TEXT),
        Property(name="document_type", data_type=DataType.TEXT),
        Property(name="business_unit", data_type=DataType.TEXT),
        Property(name="created_date", data_type=DataType.DATE),
        Property(name="tags", data_type=DataType.TEXT_ARRAY),
    ]
)

print("Schema created successfully")

Weaviate's killer feature is integrated vectorisation—simply insert text and Weaviate generates embeddings automatically. This dramatically simplifies RAG and search implementations.

Available Vectorisation Modules

Automatic Vectorisation in Action

# Insert data - vectors generated automatically
business_docs = client.collections.get("BusinessDocument")

# Single document insert
business_docs.data.insert({
    "title": "Q4 2024 Sales Report - Victoria",
    "content": """
        Quarterly sales for Victoria region exceeded targets by 15%.
        Key growth areas include Melbourne CBD (+22%), Geelong (+18%),
        and regional Victoria (+12%). The automation services division
        showed particularly strong performance with 45% year-over-year growth.
    """,
    "document_type": "report",
    "business_unit": "Sales",
    "tags": ["quarterly", "victoria", "sales"]
})

# Batch insert for efficiency
with business_docs.batch.dynamic() as batch:
    for doc in documents:
        batch.add_object(properties=doc)

# Weaviate automatically:
# 1. Sends text to OpenAI for embedding
# 2. Stores both text and vector
# 3. Indexes for search

Custom Vectorisation Configuration

# Configure which properties to vectorise
from weaviate.classes.config import Configure, Property, DataType

client.collections.create(
    name="Product",
    vectorizer_config=Configure.Vectorizer.text2vec_openai(
        model="text-embedding-3-small",
        # Only vectorise specific properties
        vectorize_collection_name=False,
    ),
    properties=[
        Property(
            name="name",
            data_type=DataType.TEXT,
            vectorize_property_name=True,  # Include property name in vector
        ),
        Property(
            name="description",
            data_type=DataType.TEXT,
            vectorize_property_name=False,
        ),
        Property(
            name="sku",
            data_type=DataType.TEXT,
            skip_vectorization=True,  # Don't include in vector
        ),
        Property(
            name="price_aud",
            data_type=DataType.NUMBER,
            skip_vectorization=True,
        ),
    ]
)

Weaviate uses GraphQL for queries with special semantic search extensions. This enables powerful yet readable queries.

Basic Semantic Search

from weaviate.classes.query import MetadataQuery

business_docs = client.collections.get("BusinessDocument")

# Semantic search - find documents similar to query
results = business_docs.query.near_text(
    query="automation ROI for small business",
    limit=5,
    return_metadata=MetadataQuery(distance=True, certainty=True)
)

for doc in results.objects:
    print(f"Title: {doc.properties['title']}")
    print(f"Distance: {doc.metadata.distance}")
    print(f"Certainty: {doc.metadata.certainty}")
    print("---")

Hybrid Search (Vectors + Keywords)

# Combine semantic and keyword search
results = business_docs.query.hybrid(
    query="GST compliance automation",
    alpha=0.75,  # 0 = pure keyword, 1 = pure semantic
    limit=10,
    return_metadata=MetadataQuery(score=True, explain_score=True)
)

# Alpha tuning:
# - 0.75-0.85: Good for most semantic search use cases
# - 0.5: Equal weight to both
# - 0.25: Prioritise keyword matches (good for exact terms like ABNs)

Filtered Search

from weaviate.classes.query import Filter

# Semantic search with filters
results = business_docs.query.near_text(
    query="compliance requirements",
    filters=(
        Filter.by_property("business_unit").equal("Legal") &
        Filter.by_property("document_type").equal("policy")
    ),
    limit=10
)

# Complex filter combinations
results = business_docs.query.hybrid(
    query="financial reporting automation",
    filters=(
        (Filter.by_property("business_unit").equal("Finance") |
         Filter.by_property("business_unit").equal("Accounting")) &
        Filter.by_property("created_date").greater_than("2024-01-01")
    ),
    alpha=0.7,
    limit=20
)

Generative Search (RAG)

# Search + AI-generated response in one query
results = business_docs.generate.near_text(
    query="What automation solutions have we implemented for Victorian clients?",
    grouped_task="""
        Based on the retrieved documents, provide:
        1. A summary of automation solutions implemented
        2. Key outcomes and ROI figures
        3. Recommendations for similar clients

        Format your response for an Australian business audience.
    """,
    limit=5
)

print("Generated Response:")
print(results.generated)

print("\nSource Documents:")
for doc in results.objects:
    print(f"- {doc.properties['title']}")

Weaviate offers both managed cloud and self-hosted options. Australian businesses should consider data residency requirements when choosing.

Weaviate Cloud Services (WCS)

# Connect to Weaviate Cloud
import weaviate
from weaviate.classes.init import Auth

client = weaviate.connect_to_weaviate_cloud(
    cluster_url="https://your-cluster.weaviate.network",
    auth_credentials=Auth.api_key("your-wcs-api-key"),
    headers={
        "X-OpenAI-Api-Key": "your-openai-key"
    }
)

# WCS Benefits:
# - Managed infrastructure
# - Automatic backups
# - Monitoring and alerting
# - Multiple region options (check for AU availability)

Self-Hosted on AWS Sydney

# Kubernetes deployment for Australian data residency
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: weaviate
  namespace: ai-platform
spec:
  serviceName: weaviate
  replicas: 3
  selector:
    matchLabels:
      app: weaviate
  template:
    metadata:
      labels:
        app: weaviate
    spec:
      containers:
      - name: weaviate
        image: semitechnologies/weaviate:latest
        ports:
        - containerPort: 8080
        env:
        - name: QUERY_DEFAULTS_LIMIT
          value: "25"
        - name: AUTHENTICATION_APIKEY_ENABLED
          value: "true"
        - name: AUTHENTICATION_APIKEY_ALLOWED_KEYS
          valueFrom:
            secretKeyRef:
              name: weaviate-secrets
              key: api-keys
        - name: ENABLE_MODULES
          value: "text2vec-openai,generative-openai"
        - name: DEFAULT_VECTORIZER_MODULE
          value: "text2vec-openai"
        - name: CLUSTER_HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
          requests:
            memory: "4Gi"
            cpu: "2"
          limits:
            memory: "8Gi"
            cpu: "4"
        volumeMounts:
        - name: weaviate-data
          mountPath: /var/lib/weaviate
  volumeClaimTemplates:
  - metadata:
      name: weaviate-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 100Gi

Authentication and Security

# API Key Authentication
import weaviate
from weaviate.classes.init import Auth

# Client with API key auth
client = weaviate.connect_to_local(
    auth_credentials=Auth.api_key("your-secure-api-key")
)

# OIDC Authentication for enterprise
client = weaviate.connect_to_custom(
    http_host="weaviate.your-domain.com.au",
    http_port=443,
    http_secure=True,
    grpc_host="weaviate-grpc.your-domain.com.au",
    grpc_port=443,
    grpc_secure=True,
    auth_credentials=Auth.bearer_token(
        access_token="your-jwt-token",
        expires_in=3600,
        refresh_token="your-refresh-token"
    )
)

Weaviate performance depends on proper index configuration, resource allocation, and query optimisation.

Index Configuration

from weaviate.classes.config import Configure, VectorDistances

# Configure HNSW index for your use case
client.collections.create(
    name="OptimizedCollection",
    vector_index_config=Configure.VectorIndex.hnsw(
        distance_metric=VectorDistances.COSINE,
        ef_construction=128,  # Higher = better recall, slower build
        max_connections=64,   # Higher = better recall, more memory
        ef=64,                # Runtime search quality parameter
        dynamic_ef_factor=8,  # Scales ef based on limit
        dynamic_ef_min=100,
        dynamic_ef_max=500,
    ),
    # ... rest of config
)

Property Indexing for Filters

# Index properties you filter on frequently
from weaviate.classes.config import Property, DataType

properties = [
    Property(
        name="business_unit",
        data_type=DataType.TEXT,
        index_filterable=True,   # Enable filtering
        index_searchable=True,   # Enable BM25 search
    ),
    Property(
        name="created_date",
        data_type=DataType.DATE,
        index_filterable=True,
        index_range_filters=True,  # Enable range queries
    ),
    Property(
        name="content",
        data_type=DataType.TEXT,
        index_filterable=False,  # Don't filter on long text
        index_searchable=True,   # But allow BM25 search
    ),
]

Batch Operations

# Efficient batch imports
collection = client.collections.get("BusinessDocument")

# Use dynamic batching
with collection.batch.dynamic() as batch:
    for doc in large_document_list:
        batch.add_object(properties=doc)

# Or fixed-size batching with rate limiting
with collection.batch.fixed_size(batch_size=100) as batch:
    for doc in documents:
        batch.add_object(properties=doc)

# Check for errors
if batch.failed_objects:
    for obj in batch.failed_objects:
        print(f"Failed: {obj.original_uuid} - {obj.message}")

Query Optimisation Tips

Performance Best Practices:

1. Limit Results
   - Always use limit parameter
   - Use pagination for large result sets
   - Avoid limit > 10000

2. Filter Early
   - Apply filters before vector search
   - Index properties used in filters
   - Use range filters on indexed dates

3. Select Properties
   - Only return properties you need
   - Large text fields slow down response
   - Use return_properties parameter

4. Batch Queries
   - Use batch endpoints for multiple queries
   - Implement request pooling in your app

5. Monitor Performance
   - Track query latency
   - Monitor memory usage
   - Set up alerting for degradation