{"id":57115,"date":"2025-10-06T05:18:29","date_gmt":"2025-10-06T05:18:29","guid":{"rendered":"https:\/\/www.newsbeep.com\/il\/57115\/"},"modified":"2025-10-06T05:18:29","modified_gmt":"2025-10-06T05:18:29","slug":"on-demand-visual-login-nodes-using-res-with-aws-parallel-computing-service","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/il\/57115\/","title":{"rendered":"On-demand visual login nodes \u2013 using RES with AWS Parallel Computing Service"},"content":{"rendered":"\n

\"On-demandAWS Parallel Computing Service (AWS PCS)<\/a> is a managed service that makes it easier for you to run and scale your high-performance computing (HPC) workloads and build scientific and engineering models on AWS using Slurm. You can use AWS PCS to build complete, elastic environments that integrate compute, storage, networking, and visualization tools. AWS PCS simplifies cluster operations with managed updates and built-in observability features, helping to remove the burden of maintenance.<\/p>\n

While PCS offers a familiar environment for researchers and engineers to run their compute jobs, individualized access that can scale across research projects is not always easy for every user. Login nodes need to be configured so that researchers can separately connect to the compute nodes and run jobs. Fortunately, this is something that Research and Engineering Studio on AWS (RES)<\/a> was built for.<\/p>\n

So today, we\u2019re happy to show you two methods for integrating PCS with Research and Engineering Studio on AWS (RES)<\/a>. In a later post, we\u2019ll dive into integration of storage and identity with our virtual desktops \/ login nodes and compute nodes.<\/p>\n

AWS Parallel Computing Service, meet RES <\/p>\n

RES is an open source, easy-to-use web-based portal for administrators to create and manage secure cloud-based research and engineering environments. With just a few clicks, scientists and engineers can create and connect to virtual desktops (VDIs). Linux VDI users, specifically, can take advantage of direct integration with PCS by turning their Linux VDI into a PCS login node<\/a>. You can see this integration between RES and PCS in the architecture diagram in Figure 1.<\/p>\n

\"Figure <\/p>\n

Figure 1 \u2013 RES + PCS reference architecture diagram. This diagram depicts how RES and PCS are typically deployed in a customer environment.<\/p>\n

Users log into their RES console via their corporate Active Directory credentials and federated identity.<\/p>\n

Users then go through RES access their personal VDI sessions, which can provide compute either directly or via PCS compute nodes.<\/p>\n

File storage (e.g., Amazon EFS, Amazon FSx for Lustre, and\/or Amazon FSx for NetApp ONTAP) are shared across the VDI (i.e., the login node) and compute nodes, with access managed centrally by AD and identity federation. S3 serves as a long-term data storage option.<\/p>\n

Lastly, admins deploy and manage the RES and PCS infrastructure via the various other AWS services in the diagram.<\/p>\n

Through the rest of this post, we\u2019ll describe the two below methods to transform users\u2019 VDIs into PCS login nodes:<\/p>\n

Option 1: Run a script directly from the VDI (only applies to that particular VDI session)
\n Option 2: Create a Software Stack<\/a> (i.e., AMI) that users can deploy on any VDI in RES [Recommended] <\/p>\n

For both options, we\u2019ve created a new HPC Recipe, Creating a RES-ready AWS PCS login node<\/a>. This recipe is part of the HPC Recipes Library<\/a> (a great resource, if you\u2019re unfamiliar with it until now).<\/p>\n

After you complete either of the two options, users will be able to simply connect to their VDIs via RES and directly submit Slurm jobs to their PCS clusters.<\/p>\n

Option 1: Run a script directly from the VDI (only applies to that particular VDI) <\/p>\n

If a given RES user already has a VDI deployed, they can simply run a script on the VDI rather than redeploying their VDI with a Software Stack created via option 2. The overall process and script can be accessed in our recipe under Manually enable PCS access on RES VDI instance (Optional<\/a>). An administrator is needed for this setup as this option requires creation of a new IAM policy for the associated RES Project as well as sudo rights for running the script on the VDI itself.<\/p>\n

This option is useful when a user wants to use a RES VDI session they already have provisioned, but is not scalable because running the script \u2013 even at boot \u2013 will take some time. To easily and more quickly enable integration between RES and PCS on creation of any Linux VDI, we recommend the second option where we create a RES Software Stack, or RES-ready AMI<\/a>, that any new VDI session can deploy.<\/p>\n

Option 2: Create a RES Software Stack (a RES-ready AMI) for any VDI session in RES [Recommended] <\/p>\n

Careful followers of this channel might notice that this option is very similar to the method we described in another post about Integrating RES with AWS ParallelCluster<\/a>. Doing this with AWS PCS is a lot simpler, though, because PCS is a fully-managed service and doesn\u2019t need us to configure a head node.<\/p>\n

Before you start, you\u2019ll need the Software Stack you want to use. Follow the steps in our recipe under RES-ready AMI integrated with PCS cluster<\/a> to create your Software Stack. This also allows you to pre-install RES dependencies and pre-bake software and configuration into your VDI images, improving boot times for your end users.<\/p>\n

We start by building EC2 Image Builder<\/a> components as well as the necessary Image Builder infrastructure using AWS CloudFormation. Our components retrieve the necessary dependencies for PCS and RES, like Slurm, sackd, and more. We\u2019ll integrated these components with a base AMI using an Image Builder recipe and pipeline running on our Amazon Elastic Compute Cloud (Amazon EC2) infrastructure, to produce our final, RES-ready AMI.<\/p>\n

EC2 Image Builder simplifies the process of building and maintaining a \u201cgolden image\u201d by allowing us to modify and rerun our pipeline when the environment or dependencies change (such as major updates to RES). If you want to know more about Image Builder, there\u2019s a short tutorial about it<\/a> in on the HPC Tech Shorts YouTube channel<\/a>.<\/p>\n

Once Image Builder finishes creating the RES-ready AMI, a RES admin can login to create the Software Stack in two steps:<\/p>\n

Update a RES Project<\/a> to add security groups that allow access to the PCS cluster (or create a Project<\/a> if it\u2019s your first time using RES). The PCSClusterSG must be added to any project that requires access to the PCS cluster. This can be added in the RES project Resource Configurations \u2192 Advanced Options \u2192 Add Security Groups configuration section.
\n Create a new Software Stack using the RES-ready AMI created from the Image Builder pipeline.
\n Assign the Software Stack to a project. Options -> Edit Stack -> Projects <\/p>\n

Once the software stack has been created, end-users that have access to it as part of a Project can create their own, dedicated login node virtual desktops.<\/p>\n

Virtual desktop login node in action <\/p>\n

End-users will access the login node virtual desktop in the same way they access any other virtual desktop.<\/p>\n

\"Figure <\/p>\n

Figure 2 \u2013 The Virtual Desktops section contains a VDI (LoginNode) which is a based on a login node compatible with PCS.<\/p>\n

Once end users have access to the login node VDI, they can interact with Slurm in the same way they\u2019re accustomed, but via a RES VDI session. This is just like a standalone PCS Login Node with the added benefit that end-users can launch their own login node as a VDI session any time they need one.<\/p>\n

\"Figure <\/p>\n

Figure 3 \u2013 A virtual desktop session showing examples of Slurm commands demonstrating the integration with PCS.<\/p>\n

To provide access to the same shared file storage across RES and PCS, you\u2019ll need to create a PCS launch template<\/a>. This template mounts your RES file storage (Amazon EFS or perhaps Amazon FSx for Lustre) and integrates your AD to the PCS compute nodes upon job submission.\u00a0While we will explore this topic in more detail in a later post, you can see examples of shared storage accessed from both a PCS compute node and the RES VDI \/ PCS login node in Figures 4 and 5.<\/p>\n

\"Figure <\/p>\n

Figure 4 \u2013 A terminal session on a PCS compute node showing the user shared storage directory listing<\/p>\n

\"Figure <\/p>\n

Figure 5 \u2013 A RES virtual desktop session showing the same shared storage directory accessible from the RES VDI \/ PCS Login Node<\/p>\n

Conclusion <\/p>\n

By integrating AWS PCS and Research and Engineering Studio on AWS, users can access the compute and storage they need via a simple and familiar interface.<\/p>\n

RES provides an interface to access a virtual login node desktop and shared storage, while Slurm efficiently manages the compute jobs in AWS Parallel Computing Service. Because PCS is a fully-managed service, the integration and management experience of this solution is even simpler than our previous recipe with AWS ParallelCluster. Teams and departments can quickly get-started today and provide users with the interface and resources they need for their HPC and AIML workloads.<\/p>\n","protected":false},"excerpt":{"rendered":"AWS Parallel Computing Service (AWS PCS) is a managed service that makes it easier for you to run…\n","protected":false},"author":2,"featured_media":57116,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[353,85,46,125],"class_list":{"0":"post-57115","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-computing","8":"tag-computing","9":"tag-il","10":"tag-israel","11":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/57115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/comments?post=57115"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/57115\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media\/57116"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media?parent=57115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/categories?post=57115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/tags?post=57115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}