Deploy W&B Platform on Google Cloud

Hosting W&B Server on Google Cloud.

W&B recommends fully managed deployment options such as W&B Multi-tenant Cloud or W&B Dedicated Cloud deployment types. W&B fully managed services are simple and secure to use, with minimum to no configuration required.

If you’ve determined to Self-Managed W&B Server, W&B recommends using the W&B Server Google Cloud Terraform Module to deploy the platform on Google Cloud.

The module documentation is extensive and contains all available options that can be used.

Before you start, W&B recommends that you choose one of the remote backends available for Terraform to store the State File.

The State File is the necessary resource to roll out upgrades or make changes in your deployment without recreating all components.

The Terraform Module will deploy the following mandatory components:

• VPC
• Cloud SQL for MySQL
• Cloud Storage Bucket
• Google Kubernetes Engine
• Memorystore for Redis
• KMS Crypto Key
• Load Balancer

Other deployment options can also include the following optional components:

• Memory store for Redis
• Pub/Sub messages system

Prerequisite permissions

The account that will run the terraform need to have the role roles/owner in the Google Cloud project used.

General steps

The steps on this topic are common for any deployment option covered by this documentation.

1. Prepare the development environment.

   • Install Terraform.
   • We recommend creating a Git repository with the code that will be used, but you can keep your files locally.
   • Create a project in Google Cloud Console.
   • Authenticate with Google Cloud (make sure to install gcloud before) using gcloud auth application-default login.

2. Create the terraform.tfvars file.

   The tvfars file content can be customized according to the installation type, but the minimum recommended will look like the example below.

   project_id  = "wandb-project"
   region      = "europe-west2"
   zone        = "europe-west2-a"
   namespace   = "wandb"
   license     = "xxxxxxxxxxyyyyyyyyyyyzzzzzzz"
   subdomain   = "wandb-gcp"
   domain_name = "wandb.ml"
   

   The variables defined here need to be decided before the deployment because. The namespace variable will be a string that will prefix all resources created by Terraform.

   The combination of subdomain and domain will form the FQDN that W&B will be configured. In the example above, the W&B FQDN will be wandb-gcp.wandb.ml.

3. Create the file variables.tf.

   For every option configured in the terraform.tfvars Terraform requires a correspondent variable declaration.

   variable "project_id" {
     type        = string
     description = "Project ID"
   }
   
   variable "region" {
     type        = string
     description = "Google region"
   }
   
   variable "zone" {
     type        = string
     description = "Google zone"
   }
   
   variable "namespace" {
     type        = string
     description = "Namespace prefix used for resources"
   }
   
   variable "domain_name" {
     type        = string
     description = "Domain name for accessing the Weights & Biases UI."
   }
   
   variable "subdomain" {
     type        = string
     description = "Subdomain for access the Weights & Biases UI."
   }
   
   variable "license" {
     type        = string
     description = "W&B License"
   }
   

Deployment - Recommended (~20 mins)

This is the most straightforward deployment option configuration that will create all Mandatory components and install in the Kubernetes Cluster the latest version of W&B.

1. Create the main.tf file.

   In the same directory where you created the files in the General Steps, create a file main.tf with the following content:

   provider "google" {
    project = var.project_id
    region  = var.region
    zone    = var.zone
   }
   
   provider "google-beta" {
    project = var.project_id
    region  = var.region
    zone    = var.zone
   }
   
   data "google_client_config" "current" {}
   
   provider "kubernetes" {
     host                   = "https://${module.wandb.cluster_endpoint}"
     cluster_ca_certificate = base64decode(module.wandb.cluster_ca_certificate)
     token                  = data.google_client_config.current.access_token
   }
   
   # Spin up all required services
   module "wandb" {
     source  = "wandb/wandb/google"
     version = "~> 10.0"
   
     namespace   = var.namespace
     license     = var.license
     domain_name = var.domain_name
     subdomain   = var.subdomain
   }
   
   # You'll want to update your DNS with the provisioned IP address
   output "url" {
     value = module.wandb.url
   }
   
   output "address" {
     value = module.wandb.address
   }
   
   output "bucket_name" {
     value = module.wandb.bucket_name
   }
   

2. Deploy W&B.

   To deploy W&B, execute the following commands:

   terraform init
   terraform apply -var-file=terraform.tfvars
   

Other deployment options

You can combine all three deployment options adding all configurations to the same file. The Terraform Module provides several options that can be combined along with the standard options and the minimal configuration found in Deployment - Recommended.

Manual configuration

To use a Google Cloud Storage bucket as a file storage backend for W&B, you will need to create a:

• PubSub Topic and Subscription
• Storage Bucket
• PubSub Notification

Create PubSub Topic and Subscription

Follow the procedure below to create a PubSub topic and subscription:

1. Navigate to the Pub/Sub service within the Google Cloud Console.
2. Select Create Topic and provide a name for your topic.
3. At the bottom of the page, select Create subscription. Ensure Delivery Type is set to Pull.
4. Click Create.

Make sure the service account or account that your instance is running has the pubsub.admin role on this subscription. For details, see the Google Cloud Pub/Sub access control documentation.

Create Storage Bucket

1. Navigate to the Cloud Storage Buckets page.

2. Select Create bucket and provide a name for your bucket. Ensure you choose a Standard storage class.

   Ensure that the service account or account that your instance is running has both:

   • access to the bucket you created in the previous step.
   • storage.objectAdmin role on this bucket. For details, see the Google Cloud Storage IAM permissions documentation.
   Your instance also needs the iam.serviceAccounts.signBlob permission in Google Cloud to create signed file URLs. Add Service Account Token Creator role to the service account or IAM member that your instance is running as to enable permission.

3. Enable CORS access. This can only be done using the command line. First, create a JSON file with the following CORS configuration.

   cors:
   - maxAgeSeconds: 3600
     method:
      - GET
      - PUT
        origin:
      - '<YOUR_W&B_SERVER_HOST>'
        responseHeader:
      - Content-Type
   

   Note that the scheme, host, and port of the values for the origin must match exactly.

4. Make sure you have gcloud installed, and logged into the correct Google Cloud project.

5. Next, run the following:

   gcloud storage buckets update gs://<BUCKET_NAME> --cors-file=<CORS_CONFIG_FILE>
   

Create PubSub Notification

Follow the procedure below in your command line to create a notification stream from the Storage Bucket to the Pub/Sub topic.

You must use the CLI to create a notification stream. Ensure you have gcloud installed.

1. Log into your Google Cloud project.

2. Run the following in your terminal:

   gcloud pubsub topics list  # list names of topics for reference
   gcloud storage ls          # list names of buckets for reference
   
   # create bucket notification
   gcloud storage buckets notifications create gs://<BUCKET_NAME> --topic=<TOPIC_NAME>
   

Further reference is available on the Cloud Storage website.

Configure W&B server

1. Finally, navigate to the W&B System Connections page at http(s)://YOUR-W&B-SERVER-HOST/console/settings/system.

2. Select the provider Google Cloud Storage (gcs).

3. Provide the name of the GCS bucket.

   

4. Press Update settings to apply the new settings.

Upgrade W&B Server

The Operator upgrades W&B automatically by the W&B Operator. To turn this off, you can update the user spec to override the images from the system console.

To pin the images to a specific version:

1. Access the ActiveSpec in the system console https://<W&B URL>/console/settings/advanced/spec/active.

2. Copy the components image configuration, which looks similar to:

     api:
       image:
         tag: 0.75.2
       initContainers:
         init-db:
           image:
             tag: 0.75.2
   
   

3. Paste the component configuration replacing the image tag in the UserSpec.

   chart: {}
   values:
     api:
       image:
         tag: 0.76.3
       initContainers:
         init-db:
           image:
             tag: 0.76.3
   
   

4. Click Save.

5. Access the tab Operator and click Trigger reapply.

   

The W&B Operator reconciles the configuration and pins the image to the specified version.