(4.9-on-demand-cloud-simulations)=
# 🧩 4.9 On-demand Cloud Simulations

```{contents}
:depth: 3
```

## 🔰 Tutorial

In this module, you will learn how to set up an AWS account and use cloud computing services like **AWS Lambda**, **Docker containers**, **Apptainer**, and **Prefect** to run **on-demand cloud simulations**. You will also explore how to integrate these simulations into a materials discovery campaign.

1. Set up an AWS account
2. Use AWS Lambda for serverless cloud computing
3. Run Docker containers and Apptainer for cloud simulations
4. Automate workflows with Prefect
5. Integrate cloud simulations into a materials discovery campaign

### Setting Up an AWS Account

**AWS (Amazon Web Services)** is a popular cloud computing platform that provides a wide range of cloud services, including compute, storage, and databases. To run on-demand simulations, you’ll first need to set up an AWS account.

#### Steps to Set Up an AWS Account:

1. **Sign up for AWS**:
   - Visit [AWS Signup](https://aws.amazon.com/free) and create an account. You’ll need to provide billing information, but AWS offers a free tier that includes limited usage for services like **AWS Lambda** and **EC2**.

2. **Create an IAM User**:
   - After signing up, create an **IAM User** for secure access to your AWS resources.
   - Go to **IAM Management Console** → **Users** → **Add User**, and assign the necessary permissions (e.g., for Lambda, S3, and EC2).

3. **Set Up AWS CLI**:
   - To manage AWS services from your terminal, install and configure the **AWS CLI**:

   ```bash
   pip install awscli
   aws configure
   ```

   Enter your **Access Key ID** and **Secret Access Key** from the IAM user you created.

4. **Security Best Practices**:
   - Enable Multi-Factor Authentication (MFA) for your root account and IAM users.
   - Regularly rotate access keys and passwords.
   - Use AWS CloudTrail to monitor and log API activity across your AWS infrastructure.

**Video Tutorial**: [How to Set Up an AWS Free Tier](https://www.youtube.com/watch?v=SFaSB6vgp8k&pp=ygUVYXdzIGZyZWUgdGllciBzaWduIHVw)

### AWS Lambda: Serverless Cloud Computing

**AWS Lambda** is a serverless compute service that lets you run code in response to events without provisioning or managing servers. It’s ideal for running simulations on-demand based on user requests or data changes.

#### Running Cloud Simulations with AWS Lambda:

1. **Create a Lambda Function**:
   - Go to **Lambda Console** → **Create Function**, choose **Author from scratch**, and select **Python** as the runtime.

2. **Write Simulation Code**:
   - Write the code for your simulation in Python. Here's a simple example that simulates material property calculations:

   ```python
   import json


   def lambda_handler(event, context):
       # Simulate material property calculations
       material_data = event["material_properties"]
       conductivity = material_data["conductivity"] * 0.9  # Example calculation
       hardness = material_data["hardness"] * 1.1

       return {
           "statusCode": 200,
           "body": json.dumps(
               {"optimized_conductivity": conductivity, "optimized_hardness": hardness}
           ),
       }
   ```

3. **Deploy the Function**:
   - Once your code is ready, deploy the Lambda function and trigger it using an **API Gateway**, **S3 events**, or **manual invocation**.

4. **Lambda Limitations**:
   - Execution time is limited to 15 minutes per invocation.
   - Memory allocation ranges from 128MB to 10GB.
   - Temporary disk space (/tmp) is limited to 512MB.

**Video Tutorial**: [AWS Lambda Tutorial for Beginners](https://www.youtube.com/watch?v=eOBq__h4OJ4)

### Docker Containers for Cloud Simulations

Docker containers help package your simulations with all necessary dependencies, making it easy to run them on any cloud server. This allows for reproducible and scalable simulations.

#### Steps to Run Cloud Simulations in Docker:

1. **Create a Dockerfile**:
   Define a Dockerfile for your simulation environment:

   ```dockerfile
   FROM python:3.8-slim

   WORKDIR /simulation

   COPY . /simulation

   RUN pip install -r requirements.txt

   CMD ["python", "run_simulation.py"]
   ```

2. **Build the Docker Image**:

   ```bash
   docker build -t material_simulation .
   ```

3. **Run the Simulation Locally**:

   ```bash
   docker run -d material_simulation
   ```

4. **Deploy the Docker Container on AWS**:
   You can deploy your Docker container on **AWS Fargate** or **EC2** for on-demand simulations. Fargate is a serverless compute engine for containers, while EC2 provides virtual machines.

5. **Using Docker Compose**:
   For more complex simulations involving multiple containers, consider using Docker Compose to define and run multi-container Docker applications.

**Video Tutorial**: [Docker on AWS EC2](https://www.youtube.com/watch?v=qNIniDftAcU&pp=ygURRG9ja2VyIG9uIEFXUyBFQzI%3D)

### Apptainer for Scientific Simulations

**Apptainer** (formerly **Singularity**) is a container platform tailored for high-performance computing (HPC) and scientific simulations. It’s commonly used in research environments where Docker cannot be used due to security constraints.

#### Steps to Run Simulations with Apptainer:

1. **Install Apptainer**:
   Install Apptainer on your system by following the instructions from the [official documentation](https://apptainer.org/docs/admin/main/installation.html).

2. **Create an Apptainer Container**:
   Similar to Docker, Apptainer uses definition files (`.def`) to define container environments. Here’s an example:

   ```def
   Bootstrap: docker
   From: python:3.8-slim

   %post
       pip install numpy scipy

   %runscript
       exec python run_simulation.py
   ```

3. **Build and Run the Container**:

   ```bash
   apptainer build material_simulation.sif material_simulation.def
   apptainer run material_simulation.sif
   ```

**Video Tutorial**: [Apptainer/Singularity Tutorial](https://www.youtube.com/watch?v=g0cCErlveiI&list=PLKZ9c4ONm-VkxWW98Gcn9H6WwykMiqtnF)

### Workflow Automation with Prefect

**Prefect** is a modern workflow orchestration tool that helps automate complex workflows, like running multiple cloud simulations or orchestrating data pipelines. You can use Prefect to trigger cloud simulations, handle retries, and manage dependencies.

#### Steps to Automate Cloud Simulations with Prefect:

1. **Install Prefect**:

   ```bash
   pip install -U prefect
   ```

2. **Define a Prefect Flow**:
   A **Prefect flow** defines the sequence of tasks to run. Here’s an example of running a cloud simulation as part of a workflow:

   ```python
   from prefect import task, Flow


   @task
   def run_simulation():
       print("Running cloud simulation...")
       # Simulate material discovery task here
       return "Simulation Completed"


   with Flow("material_discovery") as flow:
       run_simulation()

   flow.run()
   ```

3. **Run the Flow**:
   Run the flow locally or deploy it on **Prefect Cloud** for distributed execution.

**Video Tutorial**: [Getting Started with Prefect](https://www.youtube.com/watch?v=4yIW34WcmBQ&pp=ygUcR2V0dGluZyBTdGFydGVkIHdpdGggUHJlZmVjdA%3D%3D)

### Integrating Cloud Simulations into a Materials Discovery Campaign

In a materials discovery campaign, you might need to perform large-scale simulations to identify the best materials for specific properties. By integrating cloud simulations, you can scale your computational efforts and automate the discovery process.

#### Example: Integrating a Simulation into a Campaign

1. **Use Case**:
   Imagine you are searching for materials with optimal thermal conductivity and hardness. You have a dataset of potential materials, and you want to run simulations to find the top candidates.

2. **Simulation Setup**:
   - Load the material properties from a database (e.g., AWS S3).
   - Run the simulation using AWS Lambda or Docker containers on EC2/Fargate.
   - Use the simulation to calculate optimized properties (e.g., using machine learning models trained on historical material data).

3. **Automate the Process**:
   - Use **Prefect** to automate the workflow, including triggering simulations based on incoming data, aggregating the results, and storing them in a database.
   - Each simulation run generates predictions for a subset of materials, and the best candidates are selected for further analysis.

4. **Error Handling and Monitoring**:
   - Implement robust error handling in your simulations to manage unexpected issues.
   - Use AWS CloudWatch to monitor the performance and health of your cloud resources.
   - Set up alerts to notify you of any issues or anomalies in your simulations.

### Additional Resources

- [AWS Lambda Documentation](https://docs.aws.amazon.com/lambda/)
- [Docker Documentation](https://docs.docker.com/)
- [Apptainer Documentation](https://apptainer.org/)
- [Prefect Documentation](https://docs.prefect.io/)
- [EC2 User Guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html)

## 🚀 Quiz

::::{tab-set}
:sync-group: category

:::{tab-item} W 2024
:sync: w2024

:::

:::{tab-item} Sp/Su 2024
:sync: sp2024

https://q.utoronto.ca/courses/370070/assignments/1393632?display=full_width
:::

:::{tab-item} Sp/Su 2025
:sync: sp2025

https://q.utoronto.ca/courses/393352/assignments/1499737?display=full_width_with_nav
:::

::::

## 📄 Assignment

::::{tab-set}
:sync-group: category

:::{tab-item} W 2024
:sync: w2024

:::

:::{tab-item} Sp/Su 2024
:sync: sp2024

https://q.utoronto.ca/courses/370070/assignments/1394239?display=full_width
:::

:::{tab-item} Sp/Su 2025
:sync: sp2025

https://q.utoronto.ca/courses/393352/assignments/1499738?display=full_width_with_nav
:::

::::


## Course Completion

For those who have completed the course, thank you for your participation, and congratulations! 🥳

Once you have completed all the modules, fill out the [Certificate of Completion Request form](https://learn.utoronto.ca/help/forms-and-applications/confirmation-completion-request) through the School of Continuing Studies to receive your microcredential. This process may take up to 4-6 weeks. As a reminder, 70% is the required threshold to pass.

Please also consider filling out [the final course evaluation form](https://forms.office.com/r/uv76EQLSjc) listed on your course syllabus as well as [the learner equity survey](https://redcap.utoronto.ca/surveys/?s=77R3YNHDHX3TKRLD).

Once you have received your microcredential, consider adding the corresponding badge your LinkedIn profile to promote your accomplishment. Likewise, we would love to see a social media post from you using the `#AcMicrocourses` hashtag with the Acceleration Consortium account tagged ([`@acceleration-consortium`](https://www.linkedin.com/company/acceleration-consortium/) for LinkedIn and [`@acceleration_c`](https://x.com/acceleration_c) for X).