🧩 1.5 Logging Data

🧩 1.5 Logging Data #

🔰 Tutorial #

In this tutorial, you will learn how to upload data to a MongoDB database from a microcontroller and read data from a database using Python.

MongoDB database interface for a light-mixing database instance.

Chemistry and materials research data is precious. By making your data Findable, Accessible, Interoperable, and Reusable (FAIR), you can maximize the impact of your research. Here are two conceptual examples of what FAIR data might look like in the physical sciences:

FAIR data in materials science

Defining FAIR Data in materials science. Reproduced from https://doi.org/10.1557/s43577-023-00498-4

Writing to MongoDB Using the Data API #

For storing our data, we will be using MongoDB, a popular “NoSQL” database. It’s important to note that MongoDB is one of many choices for databases (in contrast to MongoDB as a document-based database, e.g., PostgreSQL for relational databases, which can be hosted via AWS, and Neo4j for graph-based databases). MongoDB is a document-oriented database, which means it stores data in JSON-like documents. MongoDB is a popular choice for internet-of-things (IoT) applications due to its ease of setup, use, and scalability. Unfortunately, MongoDB Atlas suddenly deprecated their Data API that allowed for direct reading and writing of data from devices like microcontrollers. Instead, we’ll use something called AWS Lambda as an intermediary. AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. It’s a nice choice for creating lightweight APIs and handling intermittent requests, which works well for our microcontroller application.

For the purposes of this tutorial, we have set up a free-tier test database through MongoDB Atlas and an AWS Lambda function. We provide the necessary credentials which would normally be kept private. To prevent potential misuse from distributing a public API key (which is generally not a good practice), we have granted only write permissions for this tutorial, and the database is configured to automatically delete entries once a certain storage threshold is reached. The companion notebook has a similar setup, where only read permissions are granted, which you will run later.

✅ Copy the following code into a new file on the microcontroller called write_mongodb.py and run the file. Note that you will need netman.py [permalink] and a file named my_secrets.py with your WiFi credentials, course ID, and the Lambda function URL (see [1] and [2]) you’ll be using. For this tutorial, a function URL is provided for you. You will need to create your own for the assignment.

my_secrets.py

SSID = "Enter your SSID here"
PASSWORD = "Enter your WiFi password here"
COURSE_ID = "sassy-robin"
LAMBDA_FUNCTION_URL = (
    "https://bbwl4guz6jqors4xcnnll6sucq0ttapg.lambda-url.us-east-2.on.aws/"
)

write_mongodb.py

import json

import requests
from my_secrets import LAMBDA_FUNCTION_URL, COURSE_ID, PASSWORD, SSID
from netman import connectWiFi

connectWiFi(SSID, PASSWORD, country="US")

document = {"course_id": COURSE_ID}

DATABASE_NAME = "test-db"
COLLECTION_NAME = "write-to-me"

payload = {
    "database": DATABASE_NAME,
    "collection": COLLECTION_NAME,
    "document": document,
}

response = requests.post(LAMBDA_FUNCTION_URL, json=payload)

if response.status_code == 200:
    response_data = response.json()
    print(response_data["message"])
else:
    print(f"Request failed with status code {response.status_code}")

The output should look something like the following (MAC address and IP address redacted, and your ObjectID will be different):

MAC address: ***
waiting for connection...
waiting for connection...
waiting for connection...
connected
ip = ***
Document inserted successfully: {'course_id': 'free-squirrel', '_id': ObjectId('a1b2c3')}

Reading from MongoDB Using PyMongo #

✅ Run the code from the companion notebook to read data from a MongoDB database. Note that this companion notebook also has a section about how to upload data via PyMongo (preferred when you’re able to install PyMongo on e.g., a Raspberry Pi 5 or a Windows machine, which wasn’t possible for the Pico W microcontroller). That way, you wouldn’t need to use AWS Lambda as an intermediate layer (preferred, if feasible).

Reading from MongoDB using PyMongo [Jupyter Notebook]

Reading Material #

✅ Read Community action on FAIR data will fuel a revolution in materials research

✅ Watch the following video about materials databases:

Additional Resources #

FAIR data principles website and manuscript
MongoDB: What is NoSQL?
MongoDB Getting Started Interactive Tutorial
PostgreSQL, a popular open-source relational database which can be hosted via AWS
Neo4j, a popular graph-based database
@sgbaird’s list of materials databases
Misc. awesome-materials-informatics (search for “database”), Materials-Databases (no longer maintained), awesome-chemistry-datasets
awesome-self-driving-labs research data management section
Create simple REST API (GET & POST method) with AWS Lambda Function URL
Access json request from python in python AWS lambda function
Insert Python Dictionary in PostgreSQL using Psycopg2

As a side note, MongoDB has a “serverless” option (i.e., pay only for what you use) that exceeds the free-tier limits and is more flexible than the shared and dedicated clusters, which may seem appealing at first. HOWEVER, costs will escalate quickly if the database is not optimized (e.g., the database is not indexed). If you decide to go with MongoDB Atlas for a project and need more than the 512 MB of free-tier storage, we recommend first considering the shared (e.g., M2, M5) and dedicated options, hosting your own MongoDB instance, or looking to other ecosystems. Beware of serverless. Looking to other ecosystems could be either a full replacement or a supplement (e.g., using AWS for storing large files and adding links to those files into MongoDB).

Tip

Upload larger files to Amazon S3 and store the universal resource identifier (URI) in MongoDB.

For example, let’s say that one step in your workflow involves capturing images, where the total storage required is on the order of GBs instead of MBs. Instead of saving your images directly to MongoDB (e.g., using GridFS or the BinData type), you can upload your image to Amazon S3 or similar and add the URI to the corresponding MongoDB document along with other data like sample name, image acquisition parameters, material composition, and processing conditions. The URI can then later be used to programatically access the file from Amazon S3.

For an example, see ac_training_lab/A1-cam/server.py.

If you’re having issues with writing to the database on your microcontroller and would like to quickly check if the public “write-to-me” database and corresponding Lambda Function URL is working, you can run this notebook.