metadata
base_model: Snowflake/snowflake-arctic-embed-m
datasets: []
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:1490
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: >-
How can I deploy the ZenML server in different environments and manage
pipelines with the new commands?
sentences:
- >-
ed to update the way they are registered in ZenML.the updated ZenML
server provides a new and improved collaborative experience. When
connected to a ZenML server, you can now share your ZenML Stacks and
Stack Components with other users. If you were previously using the
ZenML Profiles or the ZenML server to share your ZenML Stacks, you
should switch to the new ZenML server and Dashboard and update your
existing workflows to reflect the new features.
ZenML takes over the Metadata Store role
ZenML can now run as a server that can be accessed via a REST API and
also comes with a visual user interface (called the ZenML Dashboard).
This server can be deployed in arbitrary environments (local, on-prem,
via Docker, on AWS, GCP, Azure etc.) and supports user management,
workspace scoping, and more.
The release introduces a series of commands to facilitate managing the
lifecycle of the ZenML server and to access the pipeline and pipeline
run information:
zenml connect / disconnect / down / up / logs / status can be used to
configure your client to connect to a ZenML server, to start a local
ZenML Dashboard or to deploy a ZenML server to a cloud environment. For
more information on how to use these commands, see the ZenML deployment
documentation.
zenml pipeline list / runs / delete can be used to display information
and about and manage your pipelines and pipeline runs.
In ZenML 0.13.2 and earlier versions, information about pipelines and
pipeline runs used to be stored in a separate stack component called the
Metadata Store. Starting with 0.20.0, the role of the Metadata Store is
now taken over by ZenML itself. This means that the Metadata Store is no
longer a separate component in the ZenML architecture, but rather a part
of the ZenML core, located wherever ZenML is deployed: locally on your
machine or running remotely as a server.
- >-
ntainer │ service-principal │ │
┃┃ │ │ 🌀 kubernetes-cluster
│ access-token │ │ ┃
┃ │ │ 🐳 docker-registry
│ │ │ ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼───────────────────┼───────┼────────┨
┃ AWS Service Connector │ 🔶 aws │ 🔶 aws-generic │
implicit │ ✅ │ ✅ ┃
┃ │ │ 📦 s3-bucket │
secret-key │ │ ┃
┃ │ │ 🌀 kubernetes-cluster │
sts-token │ │ ┃
┃ │ │ 🐳 docker-registry │
iam-role │ │ ┃
┃ │ │ │
session-token │ │ ┃
┃ │ │ │
federation-token │ │ ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼───────────────────┼───────┼────────┨
┃ GCP Service Connector │ 🔵 gcp │ 🔵 gcp-generic │
implicit │ ✅ │ ✅ ┃
┃ │ │ 📦 gcs-bucket │
user-account │ │ ┃
┃ │ │ 🌀 kubernetes-cluster │
service-account │ │ ┃
┃ │ │ 🐳 docker-registry │
oauth2-token │ │ ┃
┃ │ │ │
impersonation │ │ ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┷━━━━━━━┷━━━━━━━━┛
- >-
tional) Which Metadata to Extract for the ArtifactOptionally, you can
override the extract_metadata() method to track custom metadata for all
artifacts saved by your materializer. Anything you extract here will be
displayed in the dashboard next to your artifacts.
src.zenml.metadata.metadata_types that are displayed in a dedicated way
in the dashboard. See
src.zenml.metadata.metadata_types.MetadataType for more details.
By default, this method will only extract the storage size of an
artifact, but you can overwrite it to track anything you wish. E.g., the
zenml.materializers.NumpyMaterializer overwrites this method to track
the shape, dtype, and some statistical properties of each np.ndarray
that it saves.
If you would like to disable artifact metadata extraction altogether,
you can set enable_artifact_metadata at either pipeline or step level
via @pipeline(enable_artifact_metadata=False) or
@step(enable_artifact_metadata=False).
Skipping materialization
Skipping materialization might have unintended consequences for
downstream tasks that rely on materialized artifacts. Only skip
materialization if there is no other way to do what you want to do.
While materializers should in most cases be used to control how
artifacts are returned and consumed from pipeline steps, you might
sometimes need to have a completely unmaterialized artifact in a step,
e.g., if you need to know the exact path to where your artifact is
stored.
An unmaterialized artifact is a
zenml.materializers.UnmaterializedArtifact. Among others, it has a
property uri that points to the unique path in the artifact store where
the artifact is persisted. One can use an unmaterialized artifact by
specifying UnmaterializedArtifact as the type in the step:
from zenml.artifacts.unmaterialized_artifact import
UnmaterializedArtifact
from zenml import step
@step
def my_step(my_artifact: UnmaterializedArtifact):
pd.DataFrame
pass
Example
- source_sentence: >-
How is the verification process different for multi-instance and
single-instance Service Connectors?
sentences:
- >-
Develop a Custom Annotator
Learning how to develop a custom annotator.
Before diving into the specifics of this component type, it is
beneficial to familiarize yourself with our general guide to writing
custom component flavors in ZenML. This guide provides an essential
understanding of ZenML's component flavor concepts.
Annotators are a stack component that enables the use of data annotation
as part of your ZenML stack and pipelines. You can use the associated
CLI command to launch annotation, configure your datasets and get stats
on how many labeled tasks you have ready for use.
Base abstraction in progress!
We are actively working on the base abstraction for the annotators,
which will be available soon. As a result, their extension is not
possible at the moment. If you would like to use an annotator in your
stack, please check the list of already available feature stores down
below.
PreviousProdigy
NextModel Registries
Last updated 15 days ago
- >-
ld be accessible to larger audiences.
TerminologyAs with any high-level abstraction, some terminology is
needed to express the concepts and operations involved. In spite of the
fact that Service Connectors cover such a large area of application as
authentication and authorization for a variety of resources from a range
of different vendors, we managed to keep this abstraction clean and
simple. In the following expandable sections, you'll learn more about
Service Connector Types, Resource Types, Resource Names, and Service
Connectors.
This term is used to represent and identify a particular Service
Connector implementation and answer questions about its capabilities
such as "what types of resources does this Service Connector give me
access to", "what authentication methods does it support" and "what
credentials and other information do I need to configure for it". This
is analogous to the role Flavors play for Stack Components in that the
Service Connector Type acts as the template from which one or more
Service Connectors are created.
For example, the built-in AWS Service Connector Type shipped with ZenML
supports a rich variety of authentication methods and provides access to
AWS resources such as S3 buckets, EKS clusters and ECR registries.
The zenml service-connector list-types and zenml service-connector
describe-type CLI commands can be used to explore the Service Connector
Types available with your ZenML deployment. Extensive documentation is
included covering supported authentication methods and Resource Types.
The following are just some examples:
zenml service-connector list-types
Example Command Output
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓
┃ NAME │ TYPE │ RESOURCE TYPES │
AUTH METHODS │ LOCAL │ REMOTE ┃
┠──────────────────────────────┼───────────────┼───────────────────────┼───────────────────┼───────┼────────┨
- >-
ing resources:
┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────┼────────────────┨
┃ 📦 s3-bucket │ s3://zenfiles ┃
┗━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
The following might help understand the difference between scopes:
the difference between a multi-instance and a multi-type Service
Connector is that the Resource Type scope is locked to a particular
value during configuration for the multi-instance Service Connector
similarly, the difference between a multi-instance and a multi-type
Service Connector is that the Resource Name (Resource ID) scope is
locked to a particular value during configuration for the
single-instance Service Connector
Service Connector Verification
When registering Service Connectors, the authentication configuration
and credentials are automatically verified to ensure that they can
indeed be used to gain access to the target resources:
for multi-type Service Connectors, this verification means checking that
the configured credentials can be used to authenticate successfully to
the remote service, as well as listing all resources that the
credentials have permission to access for each Resource Type supported
by the Service Connector Type.
for multi-instance Service Connectors, this verification step means
listing all resources that the credentials have permission to access in
addition to validating that the credentials can be used to authenticate
to the target service or platform.
for single-instance Service Connectors, the verification step simply
checks that the configured credentials have permission to access the
target resource.
The verification can also be performed later on an already registered
Service Connector. Furthermore, for multi-type and multi-instance
Service Connectors, the verification operation can be scoped to a
Resource Type and a Resource Name.
The following shows how a multi-type, a multi-instance and a
single-instance Service Connector can be verified with multiple scopes
after registration.
- source_sentence: >-
How long did it take to generate 1800+ questions from documentation chunks
using the local model on a GPU-enabled machine?
sentences:
- >-
ns, especially using the basic setup we have here.To give you an
indication of how long this process takes, generating 1800+ questions
from an equivalent number of documentation chunks took a little over 45
minutes using the local model on a GPU-enabled machine with Ollama.
You can view the generated dataset on the Hugging Face Hub here. This
dataset contains the original document chunks, the generated questions,
and the URL reference for the original document.
Once we have the generated questions, we can then pass them to the
retrieval component and check the results. For convenience we load the
data from the Hugging Face Hub and then pass it to the retrieval
component for evaluation. We shuffle the data and select a subset of it
to speed up the evaluation process, but for a more thorough evaluation
you could use the entire dataset. (The best practice of keeping a
separate set of data for evaluation purposes is also recommended here,
though we're not doing that in this example.)
@step
def retrieval_evaluation_full(
sample_size: int = 50,
) -> Annotated[float, "full_failure_rate_retrieval"]:
dataset = load_dataset("zenml/rag_qa_embedding_questions",
split="train")
sampled_dataset = dataset.shuffle(seed=42).select(range(sample_size))
total_tests = len(sampled_dataset)
failures = 0
for item in sampled_dataset:
generated_questions = item["generated_questions"]
question = generated_questions[
]
url_ending = item["filename"].split("/")[
1
]
_, _, urls = query_similar_docs(question, url_ending)
if all(url_ending not in url for url in urls):
logging.error(
f"Failed for question: {question}. Expected URL ending: {url_ending}.
Got: {urls}"
failures += 1
logging.info(f"Total tests: {total_tests}. Failures: {failures}")
failure_rate = (failures / total_tests) * 100
return round(failure_rate, 2)
- >-
😸Set up a project repository
Setting your team up for success with a project repository.
ZenML code typically lives in a git repository. Setting this repository
up correctly can make a huge impact on collaboration and getting the
maximum out of your ZenML deployment. This section walks users through
some of the options available to create a project repository with ZenML.
PreviousFinetuning LLMs with ZenML
NextConnect your git repository
Last updated 15 days ago
- >-
GCP Service Connector
Configuring GCP Service Connectors to connect ZenML to GCP resources
such as GCS buckets, GKE Kubernetes clusters, and GCR container
registries.
The ZenML GCP Service Connector facilitates the authentication and
access to managed GCP services and resources. These encompass a range of
resources, including GCS buckets, GCR container repositories, and GKE
clusters. The connector provides support for various authentication
methods, including GCP user accounts, service accounts, short-lived
OAuth 2.0 tokens, and implicit authentication.
To ensure heightened security measures, this connector always issues
short-lived OAuth 2.0 tokens to clients instead of long-lived
credentials unless explicitly configured to do otherwise. Furthermore,
it includes automatic configuration and detection of credentials locally
configured through the GCP CLI.
This connector serves as a general means of accessing any GCP service by
issuing OAuth 2.0 credential objects to clients. Additionally, the
connector can handle specialized authentication for GCS, Docker, and
Kubernetes Python clients. It also allows for the configuration of local
Docker and Kubernetes CLIs.
$ zenml service-connector list-types --type gcp
┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━┯━━━━━━━━┓
┃ NAME │ TYPE │ RESOURCE TYPES │ AUTH
METHODS │ LOCAL │ REMOTE ┃
┠───────────────────────┼────────┼───────────────────────┼──────────────────┼───────┼────────┨
┃ GCP Service Connector │ 🔵 gcp │ 🔵 gcp-generic │
implicit │ ✅ │ ✅ ┃
┃ │ │ 📦 gcs-bucket │
user-account │ │ ┃
┃ │ │ 🌀 kubernetes-cluster │
service-account │ │ ┃
┃ │ │ 🐳 docker-registry │
external-account │ │ ┃
┃ │ │ │
oauth2-token │ │ ┃
- source_sentence: How can I load and render reports in a Jupyter notebook using ZenML?
sentences:
- >-
❗Alerters
Sending automated alerts to chat services.
Alerters allow you to send messages to chat services (like Slack,
Discord, Mattermost, etc.) from within your pipelines. This is useful to
immediately get notified when failures happen, for general
monitoring/reporting, and also for building human-in-the-loop ML.
Alerter Flavors
Currently, the SlackAlerter and DiscordAlerter are the available alerter
integrations. However, it is straightforward to extend ZenML and build
an alerter for other chat services.
Alerter Flavor Integration Notes Slack slack slack Interacts with a
Slack channel Discord discord discord Interacts with a Discord channel
Custom Implementation custom Extend the alerter abstraction and provide
your own implementation
If you would like to see the available flavors of alerters in your
terminal, you can use the following command:
zenml alerter flavor list
How to use Alerters with ZenML
Each alerter integration comes with specific standard steps that you can
use out of the box.
However, you first need to register an alerter component in your
terminal:
zenml alerter register <ALERTER_NAME> ...
Then you can add it to your stack using
zenml stack register ... -al <ALERTER_NAME>
Afterward, you can import the alerter standard steps provided by the
respective integration and directly use them in your pipelines.
PreviousDevelop a Custom Step Operator
NextDiscord Alerter
Last updated 15 days ago
- >-
ry_similar_docs(
question: str,
url_ending: str,use_reranking: bool = False,
returned_sample_size: int = 5,
) -> Tuple[str, str, List[str]]:
"""Query similar documents for a given question and URL ending."""
embedded_question = get_embeddings(question)
db_conn = get_db_conn()
num_docs = 20 if use_reranking else returned_sample_size
top_similar_docs = get_topn_similar_docs(
embedded_question, db_conn, n=num_docs, include_metadata=True
if use_reranking:
reranked_docs_and_urls = rerank_documents(question, top_similar_docs)[
:returned_sample_size
urls = [doc[1] for doc in reranked_docs_and_urls]
else:
urls = [doc[1] for doc in top_similar_docs]
return (question, url_ending, urls)
We get the embeddings for the question being passed into the function
and connect to our PostgreSQL database. If we're using reranking, we get
the top 20 documents similar to our query and rerank them using the
rerank_documents helper function. We then extract the URLs from the
reranked documents and return them. Note that we only return 5 URLs, but
in the case of reranking we get a larger number of documents and URLs
back from the database to pass to our reranker, but in the end we always
choose the top five reranked documents to return.
Now that we've added reranking to our pipeline, we can evaluate the
performance of our reranker and see how it affects the quality of the
retrieved documents.
Code Example
To explore the full code, visit the Complete Guide repository and for
this section, particularly the eval_retrieval.py file.
PreviousUnderstanding reranking
NextEvaluating reranking performance
Last updated 1 month ago
- >-
n the respective artifact in the pipeline run DAG.Alternatively, if you
are running inside a Jupyter notebook, you can load and render the
reports using the artifact.visualize() method, e.g.:
from zenml.client import Client
def visualize_results(pipeline_name: str, step_name: str) -> None:
pipeline = Client().get_pipeline(pipeline=pipeline_name)
evidently_step = pipeline.last_run.steps[step_name]
evidently_step.visualize()
if __name__ == "__main__":
visualize_results("text_data_report_pipeline", "text_report")
visualize_results("text_data_test_pipeline", "text_test")
PreviousDeepchecks
NextWhylogs
Last updated 19 days ago
- source_sentence: >-
How do you deploy the Comet Experiment Tracker flavor provided by ZenML
integration?
sentences:
- >-
Comet
Logging and visualizing experiments with Comet.
The Comet Experiment Tracker is an Experiment Tracker flavor provided
with the Comet ZenML integration that uses the Comet experiment tracking
platform to log and visualize information from your pipeline steps
(e.g., models, parameters, metrics).
When would you want to use it?
Comet is a popular platform that you would normally use in the iterative
ML experimentation phase to track and visualize experiment results. That
doesn't mean that it cannot be repurposed to track and visualize the
results produced by your automated pipeline runs, as you make the
transition towards a more production-oriented workflow.
You should use the Comet Experiment Tracker:
if you have already been using Comet to track experiment results for
your project and would like to continue doing so as you are
incorporating MLOps workflows and best practices in your project through
ZenML.
if you are looking for a more visually interactive way of navigating the
results produced from your ZenML pipeline runs (e.g., models, metrics,
datasets)
if you would like to connect ZenML to Comet to share the artifacts and
metrics logged by your pipelines with your team, organization, or
external stakeholders
You should consider one of the other Experiment Tracker flavors if you
have never worked with Comet before and would rather use another
experiment tracking tool that you are more familiar with.
How do you deploy it?
The Comet Experiment Tracker flavor is provided by the Comet ZenML
integration. You need to install it on your local machine to be able to
register a Comet Experiment Tracker and add it to your stack:
zenml integration install comet -y
The Comet Experiment Tracker needs to be configured with the credentials
required to connect to the Comet platform using one of the available
authentication methods.
Authentication Methods
You need to configure the following credentials for authentication to
the Comet platform:
- >-
guration set up by the GCP CLI on your local host.The following is an
example of lifting GCP user credentials granting access to the same set
of GCP resources and services that the local GCP CLI is allowed to
access. The GCP CLI should already be configured with valid credentials
(i.e. by running gcloud auth application-default login). In this case,
the GCP user account authentication method is automatically detected:
zenml service-connector register gcp-auto --type gcp --auto-configure
Example Command Output
Successfully registered service connector `gcp-auto` with access to the
following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE
NAMES ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃ 🔵 gcp-generic │
zenml-core ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃ 📦 gcs-bucket │
gs://zenml-bucket-sl ┃
┃ │
gs://zenml-core.appspot.com ┃
┃ │
gs://zenml-core_cloudbuild ┃
┃ │
gs://zenml-datasets ┃
┃ │
gs://zenml-internal-artifact-store ┃
┃ │
gs://zenml-kubeflow-artifact-store ┃
┃ │
gs://zenml-project-time-series-bucket ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃ 🌀 kubernetes-cluster │
zenml-test-cluster ┃
┠───────────────────────┼─────────────────────────────────────────────────┨
┃ 🐳 docker-registry │
gcr.io/zenml-core ┃
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
zenml service-connector describe gcp-auto
Example Command Output
- >-
er Image Builder stack component, or the Vertex AIOrchestrator and Step
Operator. It should be accompanied by a matching set of
GCP permissions that allow access to the set of remote resources
required by the
client and Stack Component.
The resource name represents the GCP project that the connector is
authorized to
access.
📦 GCP GCS bucket (resource type: gcs-bucket)
Authentication methods: implicit, user-account, service-account,
oauth2-token,
impersonation
Supports resource instances: True
Authentication methods:
🔒 implicit
🔒 user-account
🔒 service-account
🔒 oauth2-token
🔒 impersonation
Allows Stack Components to connect to GCS buckets. When used by Stack
Components, they are provided a pre-configured GCS Python client
instance.
The configured credentials must have at least the following GCP
permissions
associated with the GCS buckets that it can access:
storage.buckets.list
storage.buckets.get
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.list
storage.objects.update
For example, the GCP Storage Admin role includes all of the required
permissions, but it also includes additional permissions that are not
required
by the connector.
If set, the resource name must identify a GCS bucket using one of the
following
formats:
GCS bucket URI: gs://{bucket-name}
GCS bucket name: {bucket-name}
[...]
────────────────────────────────────────────────────────────────────────────────
Please select a resource type or leave it empty to create a connector
that can be used to access any of the supported resource types
(gcp-generic, gcs-bucket, kubernetes-cluster, docker-registry). []:
gcs-bucket
Would you like to attempt auto-configuration to extract the
authentication configuration from your local environment ? [y/N]: y
Service connector auto-configured successfully with the following
configuration:
Service connector 'gcp-interactive' of type 'gcp' is 'private'.
'gcp-interactive' gcp Service
Connector Details
┏━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━┓
model-index:
- name: zenml/finetuned-snowflake-arctic-embed-m
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 384
type: dim_384
metrics:
- type: cosine_accuracy@1
value: 0.28313253012048195
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.572289156626506
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.6807228915662651
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.8012048192771084
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.28313253012048195
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.19076305220883527
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.13614457831325297
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.08012048192771083
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.28313253012048195
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.572289156626506
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.6807228915662651
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.8012048192771084
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5407472416922913
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.45774765729585015
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.46523155503040436
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 256
type: dim_256
metrics:
- type: cosine_accuracy@1
value: 0.29518072289156627
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.6024096385542169
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.6807228915662651
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7951807228915663
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.29518072289156627
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.2008032128514056
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.13614457831325297
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.0795180722891566
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.29518072289156627
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.6024096385542169
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.6807228915662651
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7951807228915663
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.5458001676537428
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.46605230445591894
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.4728738562350596
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 128
type: dim_128
metrics:
- type: cosine_accuracy@1
value: 0.2469879518072289
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.5843373493975904
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.6265060240963856
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.7409638554216867
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.2469879518072289
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.19477911646586343
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.12530120481927706
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.07409638554216866
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.2469879518072289
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.5843373493975904
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.6265060240963856
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.7409638554216867
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.4994853551416632
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.421791929623255
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.4323899020969096
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: dim 64
type: dim_64
metrics:
- type: cosine_accuracy@1
value: 0.23493975903614459
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.5
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.5783132530120482
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.6927710843373494
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.23493975903614459
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.16666666666666666
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.11566265060240961
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.06927710843373491
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.23493975903614459
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.5
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.5783132530120482
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.6927710843373494
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.4607453075643617
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.38742589405240024
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.3969546791348258
name: Cosine Map@100
zenml/finetuned-snowflake-arctic-embed-m
This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: Snowflake/snowflake-arctic-embed-m
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 768 tokens
- Similarity Function: Cosine Similarity
- Language: en
- License: apache-2.0
Model Sources
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("zenml/finetuned-snowflake-arctic-embed-m")
sentences = [
'How do you deploy the Comet Experiment Tracker flavor provided by ZenML integration?',
"Comet\n\nLogging and visualizing experiments with Comet.\n\nThe Comet Experiment Tracker is an Experiment Tracker flavor provided with the Comet ZenML integration that uses the Comet experiment tracking platform to log and visualize information from your pipeline steps (e.g., models, parameters, metrics).\n\nWhen would you want to use it?\n\nComet is a popular platform that you would normally use in the iterative ML experimentation phase to track and visualize experiment results. That doesn't mean that it cannot be repurposed to track and visualize the results produced by your automated pipeline runs, as you make the transition towards a more production-oriented workflow.\n\nYou should use the Comet Experiment Tracker:\n\nif you have already been using Comet to track experiment results for your project and would like to continue doing so as you are incorporating MLOps workflows and best practices in your project through ZenML.\n\nif you are looking for a more visually interactive way of navigating the results produced from your ZenML pipeline runs (e.g., models, metrics, datasets)\n\nif you would like to connect ZenML to Comet to share the artifacts and metrics logged by your pipelines with your team, organization, or external stakeholders\n\nYou should consider one of the other Experiment Tracker flavors if you have never worked with Comet before and would rather use another experiment tracking tool that you are more familiar with.\n\nHow do you deploy it?\n\nThe Comet Experiment Tracker flavor is provided by the Comet ZenML integration. You need to install it on your local machine to be able to register a Comet Experiment Tracker and add it to your stack:\n\nzenml integration install comet -y\n\nThe Comet Experiment Tracker needs to be configured with the credentials required to connect to the Comet platform using one of the available authentication methods.\n\nAuthentication Methods\n\nYou need to configure the following credentials for authentication to the Comet platform:",
"er Image Builder stack component, or the Vertex AIOrchestrator and Step Operator. It should be accompanied by a matching set of\n\nGCP permissions that allow access to the set of remote resources required by the\n\nclient and Stack Component.\n\nThe resource name represents the GCP project that the connector is authorized to\n\naccess.\n\n📦 GCP GCS bucket (resource type: gcs-bucket)\n\nAuthentication methods: implicit, user-account, service-account, oauth2-token,\n\nimpersonation\n\nSupports resource instances: True\n\nAuthentication methods:\n\n🔒 implicit\n\n🔒 user-account\n\n🔒 service-account\n\n🔒 oauth2-token\n\n🔒 impersonation\n\nAllows Stack Components to connect to GCS buckets. When used by Stack\n\nComponents, they are provided a pre-configured GCS Python client instance.\n\nThe configured credentials must have at least the following GCP permissions\n\nassociated with the GCS buckets that it can access:\n\nstorage.buckets.list\n\nstorage.buckets.get\n\nstorage.objects.create\n\nstorage.objects.delete\n\nstorage.objects.get\n\nstorage.objects.list\n\nstorage.objects.update\n\nFor example, the GCP Storage Admin role includes all of the required\n\npermissions, but it also includes additional permissions that are not required\n\nby the connector.\n\nIf set, the resource name must identify a GCS bucket using one of the following\n\nformats:\n\nGCS bucket URI: gs://{bucket-name}\n\nGCS bucket name: {bucket-name}\n\n[...]\n\n────────────────────────────────────────────────────────────────────────────────\n\nPlease select a resource type or leave it empty to create a connector that can be used to access any of the supported resource types (gcp-generic, gcs-bucket, kubernetes-cluster, docker-registry). []: gcs-bucket\n\nWould you like to attempt auto-configuration to extract the authentication configuration from your local environment ? [y/N]: y\n\nService connector auto-configured successfully with the following configuration:\n\nService connector 'gcp-interactive' of type 'gcp' is 'private'.\n\n'gcp-interactive' gcp Service\n\nConnector Details\n\n┏━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━┓",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
Evaluation
Metrics
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2831 |
| cosine_accuracy@3 |
0.5723 |
| cosine_accuracy@5 |
0.6807 |
| cosine_accuracy@10 |
0.8012 |
| cosine_precision@1 |
0.2831 |
| cosine_precision@3 |
0.1908 |
| cosine_precision@5 |
0.1361 |
| cosine_precision@10 |
0.0801 |
| cosine_recall@1 |
0.2831 |
| cosine_recall@3 |
0.5723 |
| cosine_recall@5 |
0.6807 |
| cosine_recall@10 |
0.8012 |
| cosine_ndcg@10 |
0.5407 |
| cosine_mrr@10 |
0.4577 |
| cosine_map@100 |
0.4652 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2952 |
| cosine_accuracy@3 |
0.6024 |
| cosine_accuracy@5 |
0.6807 |
| cosine_accuracy@10 |
0.7952 |
| cosine_precision@1 |
0.2952 |
| cosine_precision@3 |
0.2008 |
| cosine_precision@5 |
0.1361 |
| cosine_precision@10 |
0.0795 |
| cosine_recall@1 |
0.2952 |
| cosine_recall@3 |
0.6024 |
| cosine_recall@5 |
0.6807 |
| cosine_recall@10 |
0.7952 |
| cosine_ndcg@10 |
0.5458 |
| cosine_mrr@10 |
0.4661 |
| cosine_map@100 |
0.4729 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.247 |
| cosine_accuracy@3 |
0.5843 |
| cosine_accuracy@5 |
0.6265 |
| cosine_accuracy@10 |
0.741 |
| cosine_precision@1 |
0.247 |
| cosine_precision@3 |
0.1948 |
| cosine_precision@5 |
0.1253 |
| cosine_precision@10 |
0.0741 |
| cosine_recall@1 |
0.247 |
| cosine_recall@3 |
0.5843 |
| cosine_recall@5 |
0.6265 |
| cosine_recall@10 |
0.741 |
| cosine_ndcg@10 |
0.4995 |
| cosine_mrr@10 |
0.4218 |
| cosine_map@100 |
0.4324 |
Information Retrieval
| Metric |
Value |
| cosine_accuracy@1 |
0.2349 |
| cosine_accuracy@3 |
0.5 |
| cosine_accuracy@5 |
0.5783 |
| cosine_accuracy@10 |
0.6928 |
| cosine_precision@1 |
0.2349 |
| cosine_precision@3 |
0.1667 |
| cosine_precision@5 |
0.1157 |
| cosine_precision@10 |
0.0693 |
| cosine_recall@1 |
0.2349 |
| cosine_recall@3 |
0.5 |
| cosine_recall@5 |
0.5783 |
| cosine_recall@10 |
0.6928 |
| cosine_ndcg@10 |
0.4607 |
| cosine_mrr@10 |
0.3874 |
| cosine_map@100 |
0.397 |
Training Details
Training Dataset
Unnamed Dataset
- Size: 1,490 training samples
- Columns:
positive and anchor
- Approximate statistics based on the first 1000 samples:
|
positive |
anchor |
| type |
string |
string |
| details |
- min: 10 tokens
- mean: 21.23 tokens
- max: 64 tokens
|
- min: 21 tokens
- mean: 378.14 tokens
- max: 512 tokens
|
- Samples:
| positive |
anchor |
How can I organize key-value pairs into cards in the ZenML dashboard? |
Group metadata
Learn how to group key-value pairs in the dashboard.
When logging metadata passing a dictionary of dictionaries in the metadata parameter will group the metadata into cards in the ZenML dashboard. This feature helps organize metadata into logical sections, making it easier to visualize and understand.
Here's an example of grouping metadata into cards:
from zenml.metadata.metadata_types import StorageSize
log_artifact_metadata(
metadata={
"model_metrics": {
"accuracy": 0.95,
"precision": 0.92,
"recall": 0.90
},
"data_details": {
"dataset_size": StorageSize(1500000),
"feature_columns": ["age", "income", "score"]
In the ZenML dashboard, "model_metrics" and "data_details" would appear as separate cards, each containing their respective key-value pairs.
PreviousAttach metadata to steps
NextSpecial Metadata Types
Last updated 19 days ago |
How can I configure an S3 Artifact Store using implicit AWS authentication on my local machine? |
other remote stack components also running in AWS.This method uses the implicit AWS authentication available in the environment where the ZenML code is running. On your local machine, this is the quickest way to configure an S3 Artifact Store. You don't need to supply credentials explicitly when you register the S3 Artifact Store, as it leverages the local credentials and configuration that the AWS CLI stores on your local machine. However, you will need to install and set up the AWS CLI on your machine as a prerequisite, as covered in the AWS CLI documentation, before you register the S3 Artifact Store.
Certain dashboard functionality, such as visualizing or deleting artifacts, is not available when using an implicitly authenticated artifact store together with a deployed ZenML server because the ZenML server will not have permission to access the filesystem.
The implicit authentication method also needs to be coordinated with other stack components that are highly dependent on the Artifact Store and need to interact with it directly to work. If these components are not running on your machine, they do not have access to the local AWS CLI configuration and will encounter authentication failures while trying to access the S3 Artifact Store:
Orchestrators need to access the Artifact Store to manage pipeline artifacts
Step Operators need to access the Artifact Store to manage step-level artifacts
Model Deployers need to access the Artifact Store to load served models
To enable these use-cases, it is recommended to use an AWS Service Connector to link your S3 Artifact Store to the remote S3 bucket.
To set up the S3 Artifact Store to authenticate to AWS and access an S3 bucket, it is recommended to leverage the many features provided by the AWS Service Connector such as auto-configuration, best security practices regarding long-lived credentials and fine-grained access control and reusing the same credentials across multiple stack components. |
How does the sync_new_data_to_label_studio step ensure that new annotations remain in sync with the ZenML artifact store? |
th the appropriate label config with Label Studio.get_labeled_data step - This step will get all labeled data available for a particular dataset. Note that these are output in a Label Studio annotation format, which will subsequently be converted into a format appropriate for your specific use case.
sync_new_data_to_label_studio step - This step is for ensuring that ZenML is handling the annotations and that the files being used are stored and synced with the ZenML artifact store. This is an important step as part of a continuous annotation workflow since you want all the subsequent steps of your workflow to remain in sync with whatever new annotations are being made or have been created.
Helper Functions
Label Studio requires the use of what it calls 'label config' when you are creating/registering your dataset. These are strings containing HTML-like syntax that allow you to define a custom interface for your annotation. ZenML provides three helper functions that will construct these label config strings in the case of object detection, image classification, and OCR. See the integrations.label_studio.label_config_generators module for those three functions.
PreviousArgilla
NextPigeon
Last updated 15 days ago |
- Loss:
MatryoshkaLoss with these parameters:{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
384,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1
],
"n_dims_per_step": -1
}
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy: epochper_device_train_batch_size: 32per_device_eval_batch_size: 16gradient_accumulation_steps: 16learning_rate: 2e-05num_train_epochs: 4lr_scheduler_type: cosinewarmup_ratio: 0.1bf16: Truetf32: Trueload_best_model_at_end: Trueoptim: adamw_torch_fusedbatch_sampler: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir: Falsedo_predict: Falseeval_strategy: epochprediction_loss_only: Trueper_device_train_batch_size: 32per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 16eval_accumulation_steps: Nonelearning_rate: 2e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 4max_steps: -1lr_scheduler_type: cosinelr_scheduler_kwargs: {}warmup_ratio: 0.1warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Truefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Truelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Trueremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torch_fusedoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Falsehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseeval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters: auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Nonedispatch_batches: Nonesplit_batches: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falsebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportional
Training Logs
| Epoch |
Step |
dim_128_cosine_map@100 |
dim_256_cosine_map@100 |
dim_384_cosine_map@100 |
dim_64_cosine_map@100 |
| 0.6667 |
1 |
0.3939 |
0.3945 |
0.4088 |
0.2782 |
| 2.0 |
3 |
0.4298 |
0.4580 |
0.4630 |
0.3868 |
| 2.6667 |
4 |
0.4324 |
0.4729 |
0.4652 |
0.397 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.10.14
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.3.1+cu121
- Accelerate: 0.31.0
- Datasets: 2.19.1
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
MatryoshkaLoss
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
MultipleNegativesRankingLoss
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
