Valohai Onboarding

Valohai Onboarding 🦈🚀


Goals 🎯

  • Understand core Valohai concepts
  • Create and compare executions
  • Notebooks integration
  • Collect metadata
  • Manage input/output data

A Valohai execution (docs)

Create an execution 🚀

Hello Valohai 👋

pip install valohai-utils pip install jupyhai # Notebooks jupyhai install vh login vh project create vh --help
# valohai.yaml configuration file - step: name: Preprocess dataset (MNIST) image: tensorflow/tensorflow:1.13.1-py3 command: - pip install -r requirements.txt - python parameters: - name: learning_rate type: float default: 0.001 inputs: - name: training-set-images default: s3://bucket/train-images-ubyte.gz - name: training-set-labels default: s3://bucket/train-labels/* - name: test-set-images default: s3://bucket/test-images/*.jpeg - name: test-set-labels default: s3://bucket/test-labels/* keep-directories: suffix

Python example (docs)

import tensorflow as tf import numpy import valohai valohai.prepare(step='train-model', image='tensorflow/tensorflow:2.4.1') mnist = tf.keras.datasets.mnist mnist_file_path = 'mnist.npz' with numpy.load(mnist_file_path, allow_pickle=True) as f: x_train, y_train = f['x_train'], f['y_train'] x_test, y_test = f['x_test'], f['y_test'] x_train, x_test = x_train / 255.0, x_test / 255.0 model = tf.keras.models.Sequential([ tf.keras.layers.Flatten(input_shape=(28, 28)), tf.keras.layers.Dense(128, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(10) ]) predictions = model(x_train[:1]).numpy() predictions tf.nn.softmax(predictions).numpy() loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) loss_fn(y_train[:1], predictions).numpy() model.compile(optimizer='adam', loss=loss_fn, metrics=['accuracy']), y_train, epochs=5) save_path = valohai.outputs().path('model.h5')

Notebooks (docs)

Metadata and graphs (docs)

# Pipeline definition in valohai.yaml ... - pipeline: name: Train and evaluate nodes: - name: preprocess-node type: execution step: preprocess - name: train-node type: task step: train-model - name: eval-node type: execution step: evaluate edges: - [preprocess-node.output.*,] - [train-node.output.model*, eval-node.input.model]

Notebooks (docs)

# Install the tools pip install valohai-utils pip install jupyhai jupyhai init # Login and create a project mkdir valohai-sample cd valohai-sample vh login vh project create