Data scientists, DevOps/MLOps engineers
Within Domino workspaces and jobs, programmatic access to the Domino Experiment Manager (MLflow) is automatic and seamless:
from mlflow import MlflowClient
client = MlflowClient()
# use the client
Domino manages authentication and authorization transparently through a localhost reverse proxy. The proxy injects the user’s identity token and the current run ID, linking each MLflow operation to both the authenticated user and the Domino project context.
Access control is enforced at the project level:
However, many customers need to interact with the Domino Experiment Manager from outside the Domino platform. Typical scenarios include:
This document defines the supported approach for securely accessing Domino-hosted MLflow (Experiment Manager) from external clients.
To support secure access from external clients, it’s important to understand how Domino authenticates workloads to the backend MLflow service.
1. Domino-aware MLflow objects
Domino extends MLflow by attaching internal metadata tags to every experiment, run, and model. Each object is tagged with:
These tags are managed exclusively by the Domino platform and cannot be modified or overridden by user code. All access to MLflow objects is therefore mediated by the user’s project membership:
2. Enforcement architecture
Domino enforces this security model through a Domino MLflow proxy layer:
The localhost proxy injects two critical headers with each request:
The Domino MLflow proxy uses these values to verify the caller’s collaborator role for that project and apply the appropriate access control.
3. Tag integrity
When an experiment, run, or model is created, Domino automatically applies Domino-prefixed tags that encode project and ownership data. Any attempt to update or overwrite these tags is blocked by the proxy layer. This guarantees the immutability of Domino’s access control metadata and preserves the security boundary around all MLflow objects.
With the authentication model understood, we can now enable external clients to interact securely with the Domino Experiment Manager (MLflow).
Every call to the Domino MLflow proxy must include two identifiers:
DOMINO_API_KEY.DOMINO_API_KEY.To generate a valid Run ID, the client can trigger a lightweight Domino job — such as one that simply echoes a string. The job execution yields a run-id, which ties subsequent MLflow API calls to the correct project and confirms that the user (or service account) holds collaborator privileges in that project.
This approach ensures that all MLflow operations initiated externally remain subject to Domino’s standard project-level authorization and tracking mechanisms.
At the implementation level we need two request headers for every call that is made to the backend Domino MLflow proxy:
X-Domino-Api-Key — This establishes the user identityX-Domino-Execution — This is a jwt that encapsulates a Domino run-id which identifies the Domino project this MLflow interaction is for.The following Python client will demonstrate how to connect to MLflow from outside the Domino platform.
You need to configure the virtual environment. Run the following commands:
python -m venv venvsource venv/bin/activatevenv\Scripts\activatepip install mlflow==2.18.0 dominodatalab pandas scikit-learn PyJWTNote the version of MLflow (2.18.0). This should match the version of MLflow that your version of Domino is associated with.
If you have a DOMINO_USER_API_KEY you should set the environment variable. If you are approaching this as a service account user, you should bring the service account token and set the environment variable DOMINO_OAUTH_TOKEN
export DOMINO_USER_API_KEY=<DOMINO_USER_API_KEY>export DOMINO_OAUTH_TOKEN=<SERVICE_ACCOUNT_TOKEN>If the DOMINO_USER_API_KEY is not set, this client will use the DOMINO_OAUTH_TOKEN to generate a DOMINO_USER_API_KEY to pass to the MLflow client.
Next you can execute the program as follows:
python3 run_experiment_and_register_model.py \--domino_host_name "mydomino.cs.domino.tech" \--domino_project_owner "wadkars" \--domino_project_name "ddl-end-to-end-demo" \--domino_experiment_name "sw-external-ddl-end-to-end-demo" \--domino_run_name_prefix "external-run" \--domino_model_name "sw-external-ddl-end-to-end-demo"The program will create/set an experiment with the name sw-external-ddl-end-to-end-demo in the project ddl-end-to-end-demo owned by the user wadkars. It will train a model in a run with prefix external-run and model name sw-external-ddl-end-to-end-demo. Finally it will execute the “latest” version of this model by downloading it from the Domino Model Registry.
Let us look at each step:
1. As mentioned earlier you’ll need a DOMINO_USER_API_KEY. If you are using service accounts you need to generate one. The following lines of code in the script will do that:
api_key = domino_utils.get_domino_user_api_key(domino_host_name,oauth_token)
os.environ['DOMINO_USER_API_KEY'] = api_key2. Next, you’ll need a Domino workload run identifier which belongs to the project and owned by the user represented by the DOMINO_USER_API_KEY
#domino_tracking_uri is the Domino external urlresult = start_stub_job(domino_tracking_uri,project_owner,project_name,api_key,job_name)run_id=result['runId']os.environ['DOMINO_RUN_ID'] = run_id3. Now you’ll need to register the request headers for the MLflow client.
#This line is only necessary if you are encapsulating#this capability in an endpoint serving multiple requests to#MLflow from multiple principals. It clears the current thread#of the values for previous call
_request_header_provider_registry._registry.clear()
#See the file domino_api_key_header_provider.pyregister_domino_api_key_request_header_provider()
#See the file domino_execution_header_provider.pyregister_domino_execution_request_header_provider()And you are all set! Simply access MLflow from the external client as if you were inside a Domino workload.
start_run_and_register_model(experiment_name,run_name_prefix,model_name)result = download_and_run_logged_model(model_name,"latest",[[0,1]])print(result)If you read through the implementation of these methods, you will notice that it is simply vanilla MLflow code.
Tracking experiments executed outside Domino requires explicit project context. Because every MLflow object must be associated with a Domino project, external work must be logged against a surrogate Domino project that serves as its system-of-record anchor.
Recommended setup:
This approach preserves Domino’s access-control semantics and ensures that experiments, runs, and models created by external clients remain governed by the same project-level authorization rules as internal Domino workloads.
What are the alternatives to using the above method? The obvious one is to keep all MLflow interaction inside a Domino job. The external client will simply invoke a job using the Domino python client.
This approach should be preferred if you are running a training or a scoring job inside Domino.
However it does not work for the two of the three key use-cases we described earlier:
One last note — Domino is planning to deprecate API keys at a later date. This article will be updated to reflect that change when it happens.
Principal solution architect
I work closely with enterprise customers to deeply understand their environments and enable successful adoption of the Domino platform. I've designed and delivered solutions that address real-world challenges, with several becoming part of the core product. My focus is on scalable infrastructure for LLM inference, distributed training, and secure cloud-to-edge deployments, bridging advanced machine learning with operational needs.
