01
PureML-version
Manage versioning of datasets and models with our python SDK. Versioning is semantic and automatically managed.
Install
Getting started is simple
$ pip install puremlDataset
Simply use our decorator @dataset for managing the versions of your dataset.
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from pureml.decorators import dataset
@dataset("petdata:dev")
def load_data(img_folder = "PetImages"):
image_size = (180, 180)
batch_size = 16
train_ds,
val_ds = tf.keras.utils.img_dataset_from_directory(
img_folder,
validation_split=0.2,
subset="both",
seed=1337,
image_size=image_size,
batch_size=batch_size,
)
data_augmentation = keras.Sequential(
[layers.RandomFlip("horizontal"),
layers.RandomRotation(0.1),]
)
train_ds = train_ds.map(
lambda img, label: (data_augmentation(img), label),
num_parallel_calls=tf.data.AUTOTUNE,
)
train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
val_ds = val_ds.prefetch(tf.data.AUTOTUNE)
return train_ds, val_ds
Key-value pairs
Our system captures key-value metadata such as metrics and associates it with the version of the model.
Large files
Our ML versioning system is built to natively support large files, unlike Git.
Powerful yet flexible
Our SDK is designed to be user-friendly, yet robust enough to meet a wide range of use cases.
02
PureML-eval : Testing & Quality Control
The terms L0, L1, L2, L3, and L4 refer to teams involved in developing machine learning systems. Our research study which surveyed 100 companies found that the majority of teams are currently at the L0 level. However, some teams are striving to build highly dependable systems and are at L3 or L4 levels. We are developing tools for the teams to facilitate their journey to attain the L4 level.
Coming soon...
Step a.1: Use an existing model for validation
import pureml
pureml.dataset.validation(“petdata:dev:v1”)Use @validationfor adding a dataset as validation while saving it.
This helps us capture not just one instance of this dataset, but all the future variations without any intervention.
Step a.2: Register validation dataset
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from pureml.decorators import dataset, validation
@validation
@dataset("petdata:dev")
def load_data(img_folder = "PetImages"):
image_size = (180, 180)
batch_size = 16
train_ds,
val_ds = tf.keras.utils.img_dataset_from_directory(
img_folder,
validation_split=0.2,
subset="both",
seed=1337,
image_size=image_size,
batch_size=batch_size,
)
data_augmentation = keras.Sequential(
[
layers.RandomFlip("horizontal"),
layers.RandomRotation(0.1),
]
)
train_ds = train_ds.map(
lambda img, label: (data_augmentation(img), label),
num_parallel_calls=tf.data.AUTOTUNE,
)
train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
val_ds = val_ds.prefetch(tf.data.AUTOTUNE)
return train_ds, val_dsStep b: Predictor for model
We recommend utilizing our base predictor class when developing your model. By doing so, you can leverage the predict function in this class as your model's prediction function, which can be used in various stages such as testing, inference, and dockerization.
from pureml import BasePredictor
import pureml
import tensorflow as tf
from tensorflow import keras
class Predictor(BasePredictor):
model_details = ['pet_classifier:dev:latest']
input={'type': 'image'},
output={'type': 'numpy ndarray' }
def load_models(self):
self.model = pureml.model.fetch(self.model_details)
def predict(self, pred_img):
pred_img = keras.preprocessing.image.img_to_array(
pred_img
)
pred_img = tf.expand_dims(pred_img, 0)
predictions = self.model.predict(pred_img)
predictions = float(predictions[0])
return predictionsStep c: Evaluating your model is done as follows
import pureml
pureml.model.evaluate("pet_classifier:dev:v1", "petdata:dev:v1")03
Review
By providing a comprehensive set of metrics and visualizations, PureML makes it easy to identify and correct any issues with its review feature and allows you to evaluate the quality of their data and the accuracy of their model.
04
PureML-package
PureML is a versatile tool that allows you to package your machine learning models into a standard, production-ready container. Additionally, you can utilize a user-friendly web interface to demonstrate your machine learning model, making it easily accessible to anyone, from anywhere.
pureml.docker.create(“pet_classifier:dev:v1”)05
PureML-deploy
PureML gives you the ability to deploy machine learning models without the need for managing infrastructure or servers.
$ pureml deploy pet_classifier:dev:v1
