🧩 4.9 On-demand Cloud Simulations#
🔰 Tutorial#
In this module, you will learn how to set up an AWS account and use cloud computing services like AWS Lambda, Docker containers, Apptainer, and Prefect to run on-demand cloud simulations. You will also explore how to integrate these simulations into a materials discovery campaign.
Set up an AWS account
Use AWS Lambda for serverless cloud computing
Run Docker containers and Apptainer for cloud simulations
Automate workflows with Prefect
Integrate cloud simulations into a materials discovery campaign
Setting Up an AWS Account#
AWS (Amazon Web Services) is a popular cloud computing platform that provides a wide range of cloud services, including compute, storage, and databases. To run on-demand simulations, you’ll first need to set up an AWS account.
Steps to Set Up an AWS Account:#
Sign up for AWS:
Visit AWS Signup and create an account. You’ll need to provide billing information, but AWS offers a free tier that includes limited usage for services like AWS Lambda and EC2.
Create an IAM User:
After signing up, create an IAM User for secure access to your AWS resources.
Go to IAM Management Console → Users → Add User, and assign the necessary permissions (e.g., for Lambda, S3, and EC2).
Set Up AWS CLI:
To manage AWS services from your terminal, install and configure the AWS CLI:
pip install awscli aws configure
Enter your Access Key ID and Secret Access Key from the IAM user you created.
Security Best Practices:
Enable Multi-Factor Authentication (MFA) for your root account and IAM users.
Regularly rotate access keys and passwords.
Use AWS CloudTrail to monitor and log API activity across your AWS infrastructure.
Video Tutorial: How to Set Up an AWS Free Tier
AWS Lambda: Serverless Cloud Computing#
AWS Lambda is a serverless compute service that lets you run code in response to events without provisioning or managing servers. It’s ideal for running simulations on-demand based on user requests or data changes.
Running Cloud Simulations with AWS Lambda:#
Create a Lambda Function:
Go to Lambda Console → Create Function, choose Author from scratch, and select Python as the runtime.
Write Simulation Code:
Write the code for your simulation in Python. Here’s a simple example that simulates material property calculations:
import json def lambda_handler(event, context): # Simulate material property calculations material_data = event["material_properties"] conductivity = material_data["conductivity"] * 0.9 # Example calculation hardness = material_data["hardness"] * 1.1 return { "statusCode": 200, "body": json.dumps( {"optimized_conductivity": conductivity, "optimized_hardness": hardness} ), }
Deploy the Function:
Once your code is ready, deploy the Lambda function and trigger it using an API Gateway, S3 events, or manual invocation.
Lambda Limitations:
Execution time is limited to 15 minutes per invocation.
Memory allocation ranges from 128MB to 10GB.
Temporary disk space (/tmp) is limited to 512MB.
Video Tutorial: AWS Lambda Tutorial for Beginners
Docker Containers for Cloud Simulations#
Docker containers help package your simulations with all necessary dependencies, making it easy to run them on any cloud server. This allows for reproducible and scalable simulations.
Steps to Run Cloud Simulations in Docker:#
Create a Dockerfile: Define a Dockerfile for your simulation environment:
FROM python:3.8-slim WORKDIR /simulation COPY . /simulation RUN pip install -r requirements.txt CMD ["python", "run_simulation.py"]
Build the Docker Image:
docker build -t material_simulation .
Run the Simulation Locally:
docker run -d material_simulation
Deploy the Docker Container on AWS: You can deploy your Docker container on AWS Fargate or EC2 for on-demand simulations. Fargate is a serverless compute engine for containers, while EC2 provides virtual machines.
Using Docker Compose: For more complex simulations involving multiple containers, consider using Docker Compose to define and run multi-container Docker applications.
Video Tutorial: Docker on AWS EC2
Apptainer for Scientific Simulations#
Apptainer (formerly Singularity) is a container platform tailored for high-performance computing (HPC) and scientific simulations. It’s commonly used in research environments where Docker cannot be used due to security constraints.
Steps to Run Simulations with Apptainer:#
Install Apptainer: Install Apptainer on your system by following the instructions from the official documentation.
Create an Apptainer Container: Similar to Docker, Apptainer uses definition files (
.def) to define container environments. Here’s an example:Bootstrap: docker From: python:3.8-slim %post pip install numpy scipy %runscript exec python run_simulation.pyBuild and Run the Container:
apptainer build material_simulation.sif material_simulation.def apptainer run material_simulation.sif
Video Tutorial: Apptainer/Singularity Tutorial
Workflow Automation with Prefect#
Prefect is a modern workflow orchestration tool that helps automate complex workflows, like running multiple cloud simulations or orchestrating data pipelines. You can use Prefect to trigger cloud simulations, handle retries, and manage dependencies.
Steps to Automate Cloud Simulations with Prefect:#
Install Prefect:
pip install -U prefect
Define a Prefect Flow: A Prefect flow defines the sequence of tasks to run. Here’s an example of running a cloud simulation as part of a workflow:
from prefect import task, Flow @task def run_simulation(): print("Running cloud simulation...") # Simulate material discovery task here return "Simulation Completed" with Flow("material_discovery") as flow: run_simulation() flow.run()
Run the Flow: Run the flow locally or deploy it on Prefect Cloud for distributed execution.
Video Tutorial: Getting Started with Prefect
Integrating Cloud Simulations into a Materials Discovery Campaign#
In a materials discovery campaign, you might need to perform large-scale simulations to identify the best materials for specific properties. By integrating cloud simulations, you can scale your computational efforts and automate the discovery process.
Example: Integrating a Simulation into a Campaign#
Use Case: Imagine you are searching for materials with optimal thermal conductivity and hardness. You have a dataset of potential materials, and you want to run simulations to find the top candidates.
Simulation Setup:
Load the material properties from a database (e.g., AWS S3).
Run the simulation using AWS Lambda or Docker containers on EC2/Fargate.
Use the simulation to calculate optimized properties (e.g., using machine learning models trained on historical material data).
Automate the Process:
Use Prefect to automate the workflow, including triggering simulations based on incoming data, aggregating the results, and storing them in a database.
Each simulation run generates predictions for a subset of materials, and the best candidates are selected for further analysis.
Error Handling and Monitoring:
Implement robust error handling in your simulations to manage unexpected issues.
Use AWS CloudWatch to monitor the performance and health of your cloud resources.
Set up alerts to notify you of any issues or anomalies in your simulations.
Additional Resources#
🚀 Quiz#
📄 Assignment#
Course Completion#
For those who have completed the course, thank you for your participation, and congratulations! 🥳
Once you have completed all the modules, fill out the Certificate of Completion Request form through the School of Continuing Studies to receive your microcredential. This process may take up to 4-6 weeks. As a reminder, 70% is the required threshold to pass.
Please also consider filling out the final course evaluation form listed on your course syllabus as well as the learner equity survey.
Once you have received your microcredential, consider adding the corresponding badge your LinkedIn profile to promote your accomplishment. Likewise, we would love to see a social media post from you using the #AcMicrocourses hashtag with the Acceleration Consortium account tagged (@acceleration-consortium for LinkedIn and @acceleration_c for X).