Vertex AI supports custom container training by letting you bring your own Docker image that contains your framework, dependencies, and training code, then run it on managed, autoscaled infrastructure. In simple terms, you build a container that exposes an entrypoint (e.g., python train.py), push it to Artifact Registry, and submit a CustomJob specifying the container URI, machine type, accelerators (GPU/TPU if needed), and environment variables. Vertex AI handles orchestration details such as provisioning nodes, mounting Cloud Storage paths, streaming logs to Cloud Logging, wiring TensorBoard, and collecting artifacts and metrics. You can also define distributed strategies (e.g., PyTorch DDP or TensorFlow MultiWorkerMirroredStrategy) by requesting multiple workers and parameter servers or using a multi-worker container pattern.
A typical setup looks like this: data is stored in Cloud Storage or read directly from BigQuery; your container references data paths via flags or environment variables; training outputs (checkpoints, model artifacts) are written back to Cloud Storage. For reproducibility, you pin dependency versions in your Dockerfile, pass a Git SHA through an env var, and write training/validation metrics to a well-known path so Vertex can surface them in the UI. Hyperparameter tuning can wrap your CustomJob, launching parallel trials that vary learning rates, batch sizes, or architecture flags. Each trial uses the same container, ensuring consistent environments across experiments.
If your downstream application uses embeddings with Milvus, custom containers are a clean way to own the embedding generation logic. For example, you can train a domain-specific contrastive model (text-text or text-image) and export a saved model. After training, run a batch prediction job (also via a container) to embed your corpus and upsert vectors into Milvus. Keep vector dimensionality and preprocessing consistent between training and serving, store stable IDs alongside vectors, and log evaluation metrics like recall@k against a labeled query set. This separation—custom container for model logic, Vertex AI for orchestration, Milvus for retrieval—keeps the pipeline scalable and maintainable.