The Anyscale platform<\/a> integrates seamlessly with SageMaker HyperPod when using Amazon Elastic Kubernetes Service<\/a> (Amazon EKS) as the cluster orchestrator. Ray<\/a> is the leading AI compute engine, offering Python-based distributed computing capabilities to address AI workloads ranging from multimodal AI, data processing, model training, and model serving. Anyscale unlocks the power of Ray with comprehensive tooling for developer agility, critical fault tolerance, and an optimized version called RayTurbo<\/a>, designed to deliver leading cost-efficiency. Through a unified control plane, organizations benefit from simplified management of complex distributed AI use cases with fine-grained control across hardware.<\/p>\n The combined solution provides extensive monitoring through SageMaker HyperPod real-time dashboards<\/a> tracking node health, GPU utilization, and network traffic. Integration with Amazon CloudWatch Container Insights<\/a>, Amazon Managed Service for Prometheus<\/a>, and Amazon Managed Grafana<\/a> delivers deep visibility into cluster performance, complemented by Anyscale\u2019s monitoring framework<\/a>, which provides built-in metrics for monitoring Ray clusters and the workloads that run on them.<\/p>\n This post demonstrates how to integrate the Anyscale platform with SageMaker HyperPod. This combination can deliver tangible business outcomes: reduced time-to-market for AI initiatives, lower total cost of ownership through optimized resource utilization, and increased data science productivity by minimizing infrastructure management overhead. It is ideal for Amazon EKS and Kubernetes-focused organizations, teams with large-scale distributed training needs, and those invested in the Ray ecosystem<\/a> or SageMaker.<\/p>\n Solution overview <\/p>\n The following architecture diagram illustrates SageMaker HyperPod with Amazon EKS orchestration and Anyscale.<\/p>\n The sequence of events in this architecture is as follows:<\/p>\n A user submits a job to the Anyscale Control Plane, which is the main user-facing endpoint. This flow shows distribution and execution of user-submitted jobs across the available computing resources, while maintaining monitoring and data accessibility throughout the process.<\/p>\n Prerequisites <\/p>\n Before you begin, you must have the following resources:<\/p>\n Set up Anyscale Operator <\/p>\n Complete the following steps to set up the Anyscale Operator:<\/p>\n In your workspace, download the aws-do-ray<\/a> repository: <\/p>\n git clone https:\/\/github.com\/aws-samples\/aws-do-ray.git This repository has the commands needed to deploy the Anyscale Operator on a SageMaker HyperPod cluster. The aws-do-ray<\/a> project aims to simplify the deployment and scaling of distributed Python application using Ray on Amazon EKS or SageMaker HyperPod. The aws-do-ray container shell is equipped with intuitive action scripts and comes preconfigured with convenient shortcuts, which save extensive typing and increase productivity. You can optionally use these features by building and opening a bash shell in the container with the instructions in the aws-do-ray README, or you can continue with the following steps.<\/p>\n If you continue with these steps, make sure your environment is properly set up: <\/p>\n Verify your connection to the HyperPod cluster: <\/p>\n Obtain the name of the EKS cluster on the SageMaker HyperPod console. In your cluster details, you will see your EKS cluster orchestrator. aws eks update-kubeconfig –region –name my-eks-cluster<\/p>\n kubectl get nodes -L node.kubernetes.io\/instance-type -L sagemaker.amazonaws.com\/node-health-status -L sagemaker.amazonaws.com\/deep-health-check-status $@ <\/p>\n The following screenshot shows an example output.<\/p>\n If the output indicates InProgress instead of Passed, wait for the deep health checks to finish.<\/p>\n Review the env_vars file. Update the variable AWS_EKS_HYPERPOD_CLUSTER. You can leave the values as default or make desired changes. Execute: This creates the anyscale namespace, installs Anyscale dependencies, configures login to your Anyscale account (this step will prompt you for additional verification as shown in the following screenshot), adds the anyscale helm chart<\/a>, installs the ingress-nginx<\/a> controller, and finally labels and taints SageMaker HyperPod nodes for the Anyscale worker pods.<\/p>\n Create an EFS file system: <\/p>\n Execute:<\/p>\n .\/2.create-efs.sh <\/p>\n Amazon EFS serves as the shared cluster storage<\/a> for the Anyscale pods. Register an Anyscale Cloud: <\/p>\n Execute:<\/p>\n .\/3.register-cloud.sh <\/p>\n This registers a self-hosted Anyscale Cloud into your SageMaker HyperPod cluster. By default, it uses the value of ANYSCALE_CLOUD_NAME in the env_vars file. You can modify this field as needed. At this point, you will be able to see your registered cloud on the Anyscale console<\/a>.<\/p>\n Deploy the Kubernetes Anyscale Operator: <\/p>\n Execute:<\/p>\n .\/4.deploy-anyscale.sh <\/p>\n This command installs the Anyscale Operator in the anyscale namespace. The Operator will start posting health checks to the Anyscale Control Plane.<\/p>\n To see the Anyscale Operator pod, run the following command:kubectl get pods -n anyscale<\/p>\n Submit training job <\/p>\n This section walks through a simple training job submission. The example implements distributed training of a neural network for Fashion MNIST classification using the Ray Train framework on SageMaker HyperPod with Amazon EKS orchestration, demonstrating how to use the AWS managed ML infrastructure combined with Ray\u2019s distributed computing capabilities for scalable model training.Complete the following steps:<\/p>\n Navigate to the jobs directory. This contains folders for available example jobs you can run. For this walkthrough, go to the dt-pytorch directory containing the training job. <\/p>\n Configure the required environment variables: <\/p>\n AWS_ACCESS_KEY_ID Create Anyscale compute configuration<\/a>: Clean up <\/p>\n To clean up your Anyscale cloud, run the following command:<\/p>\n cd ..\/.. To delete your SageMaker HyperPod cluster and associated resources, delete the CloudFormation stack if this is how you created the cluster and its resources.<\/p>\n Conclusion <\/p>\n This post demonstrated how to set up and deploy the Anyscale Operator on SageMaker HyperPod using Amazon EKS for orchestration.SageMaker HyperPod and Anyscale RayTurbo provide a highly efficient, resilient solution for large-scale distributed AI workloads: SageMaker HyperPod delivers robust, automated infrastructure management and fault recovery for GPU clusters, and RayTurbo accelerates distributed computing and optimizes resource usage with no code changes required. By combining the high-throughput, fault-tolerant environment of SageMaker HyperPod with RayTurbo\u2019s faster data processing and smarter scheduling, organizations can train and serve models at scale with improved reliability and significant cost savings, making this stack ideal for demanding tasks like large language model pre-training and batch inference.<\/p>\n For more examples of using SageMaker HyperPod, refer to the Amazon EKS Support in Amazon SageMaker HyperPod workshop<\/a> and the Amazon SageMaker HyperPod Developer Guide<\/a>. For information on how customers are using RayTurbo, refer to RayTurbo<\/a>.<\/p>\n \u00a0<\/p>\n About the authors <\/p>\n![\"End-to-end](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-arch-diag-1024x570.png\")
<\/p>\n
\n The Anyscale Control Plane communicates this job to the Anyscale Operator within the SageMaker HyperPod cluster in the SageMaker HyperPod virtual private cloud (VPC).
\n The Anyscale Operator, upon receiving the job, initiates the process of creating the necessary pods by reaching out to the EKS control plane.
\n The EKS control plane orchestrates creation of a Ray head pod and worker pods. These pods represent a Ray cluster, running on SageMaker HyperPod with Amazon EKS.
\n The Anyscale Operator submits the job through the head pod, which serves as the primary coordinator for the distributed workload.
\n The head pod distributes the workload across multiple worker pods, as shown in the hierarchical structure in the SageMaker HyperPod EKS cluster.
\n Worker pods execute their assigned tasks, potentially accessing required data from the storage services \u2013 such as Amazon Simple Storage Service<\/a> (Amazon S3), Amazon Elastic File System<\/a> (Amazon EFS), or Amazon FSx for Lustre<\/a> \u2013 in the user VPC.
\n Throughout the job execution, metrics and logs are published to Amazon CloudWatch<\/a> and Amazon Managed Service for Prometheus or Amazon Managed Grafana for observability.
\n When the Ray job is complete, the job artifacts (final model weights, inference results, and so on) are saved to the designated storage service.
\n Job results (status, metrics, logs) are sent through the Anyscale Operator back to the Anyscale Control Plane. <\/p>\n
\ncd aws-do-ray\/Container-Root\/ray\/anyscale <\/p>\n![\"Active](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-cluster-console-1024x290.png\")
\n Update kubeconfig to connect to the EKS cluster: <\/p>\n![\"Terminal](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-example-output-1-1024x54.png\")
<\/p>\n![\"Terminal](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-example-output-2-1024x52.png\")
<\/p>\n
\n Deploy your requirements: <\/p>\n
\n.\/1.deploy-requirements.sh <\/p>\n![\"Terminal](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-additional-verification-1024x179.jpeg\")
<\/p>\n
At the time of writing, Amazon EFS and S3FS are the supported file system options when using Anyscale and SageMaker HyperPod setups with Ray on AWS. Although FSx for Lustre is not supported with this setup, you can use it with KubeRay on SageMaker HyperPod EKS.<\/p>\n
\nAWS_SECRET_ACCESS_KEY
\nAWS_REGION
\nANYSCALE_CLOUD_NAME <\/p>\n
.\/1.create-compute-config.sh
\n Submit the training job:
.\/2.submit-dt-pytorch.shThis uses the job configuration specified in job_config.yaml. For more information on the job config, refer to JobConfig<\/a>.
\n Monitor the deployment. You will see the newly created head and worker pods in the anyscale namespace.
kubectl get pods -n anyscale
\n View the job status and logs on the Anyscale console to monitor your submitted job\u2019s progress and output.![\"Ray](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-training-logs-1024x557.png\")
<\/p>\n
\n.\/5.remove-anyscale.sh <\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-sindhura.jpeg\")
Sindhura Palakodety<\/a> is a Senior Solutions Architect at AWS and Single-Threaded Leader (STL) for ISV Generative AI, where she is dedicated to empowering customers in developing enterprise-scale, Well-Architected solutions. She specializes in generative AI and data analytics domains, helping organizations use innovative technologies for transformative business outcomes.<\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-mark-1.jpeg\")
Mark Vinciguerra<\/a> is an Associate Specialist Solutions Architect at AWS based in New York. He focuses on generative AI training and inference, with the goal of helping customers architect, optimize, and scale their workloads across various AWS services. Prior to AWS, he went to Boston University and graduated with a degree in Computer Engineering.<\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-flo-1.jpeg\")
Florian Gauter<\/a> is a Worldwide Specialist Solutions Architect at AWS, based in Hamburg, Germany. He specializes in AI\/ML and generative AI solutions, helping customers optimize and scale their AI\/ML workloads on AWS. With a background as a Data Scientist, Florian brings deep technical expertise to help organizations design and implement sophisticated ML solutions. He works closely with customers worldwide to transform their AI initiatives and maximize the value of their ML investments on AWS.<\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-alex.jpeg\")
Alex Iankoulski<\/a> is a Principal Solutions Architect in the Worldwide Specialist Organization at AWS. He focuses on orchestration of AI\/ML workloads using containers. Alex is the author of the do-framework<\/a> and a Docker captain<\/a> who loves applying container technologies to accelerate the pace of innovation while solving the world\u2019s biggest challenges. Over the past 10 years, Alex has worked on helping customers do more on AWS, democratizing AI and ML, combating climate change, and making travel safer, healthcare better, and energy smarter.<\/p>\n
![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-anoop.jpeg\")
![\"\"](\"https:\/\/www.newsbeep.com\/uk\/wp-content\/uploads\/2025\/10\/ml-19194-dominic-100x103.png\")