You should use Step Functions more! (AWS)

Introduction

A while ago, I discovered the AWS Step Functions service and after just a few hours of exploration, I was totally captivated by its potential. Indeed, this is a service at AWS that is close to the “No Code” concept. Initially, I had already heard of Step Functions, but I had the impression that this service was limited to chaining AWS Lambda functions (serverless). However, after spending some time using it, I realized that it is much more than that.

Today I want to share this discovery with you and explain why you should consider using this service more frequently. I will avoid giving you specific details on its use, because my intention is rather to share with you my feedback (REX) about it.

Service overview

AWS Step Functions is a workflow management service offered by Amazon Web Services (AWS) that allows developers (XXXX) to easily coordinate and automate multi-step applications. It makes it easy to create workflows orchestrating different tasks and services without requiring complex management of the underlying infrastructure.

To better understand what workflows are, imagine a series of steps to follow to accomplish a given task. Each step may involve processing, verifications, calls to external services, etc. Workflows are commonly used to coordinate distributed operations, trigger conditional processing, and react to events.

Like “A good sketch is better than a long speech” this is what it can look like:

AWS Step Functions provides a user-friendly GUI to visually design and visualize these workflows. You can create finite state machines with different steps, conditional decisions and parallel branches. This allows you to orchestrate AWS services (like AWS Lambda, AWS Glue, AWS Batch, etc.) and custom actions in a precise order, based on specified conditions.

Trigger

A step function can be launched by:

• aws cli => manual trigger
• event bridge => triggered from an event
• build code => CICD trigger
• api gateway => API call trigger

Now that the presentations are made here are some examples

Case examples

Example use case:

1. Use case: Image processing in a medical treatment pipeline

Let’s say you work for a company that develops medical image processing solutions. You have an application that supports processing X-ray images to detect abnormalities and diseases. The processing process of each image includes several steps, such as data preparation, analysis by a machine learning model, report generation, and storage of results.

To accomplish this process efficiently and scalably, you can use AWS Step Functions. Here’s how it could work:

has. Uploading the image: The user uploads the X-ray image to a storage service, such as Amazon S3.

b. Workflow trigger: Once the image is uploaded, a trigger (for example, an S3 event) notifies AWS Step Functions to start the image processing workflow.

vs. Data preparation (State 1): In this step, the image is prepared for processing. This may include image validation, resizing, or other necessary preprocessing before submitting it to the machine learning model.

d. Image analysis (State 2): After the image is prepared, AWS Step Functions calls a Lambda function that runs the machine learning model to analyze the image and detect any anomalies or diseases.

e. Report generation (State 3): After analyzing the image, AWS Step Functions can call another Lambda function to generate a report containing the analysis results and relevant information about the image.

f. Storage of results (State 4): Finally, the analysis results and the generated report can be stored in a database or other storage service, such as Amazon DynamoDB or Amazon RDS.

Using AWS Step Functions makes it easy to manage workflow coordination and management, while ensuring the scalability and reliability of the medical image processing process. In case of an error or failure in any of the steps, Step Functions also offers the possibility to manage retries or error handling strategies, thus guaranteeing the robustness of the application.

2. Use case: A data extractor

Suppose you need to extract data from several tables in a database, transform it and send it to a team via Slack or email. You’ll use AWS Step Functions to orchestrate this process, leveraging AWS Athena to run SQL queries, S3 to store the results, Lambda to transform the data, and SNS or SES to send notifications.

a. Triggering the workflow The workflow starts with a trigger. This trigger can be manual, for example.

b. Parallel execution of several SQL requests via Athena Once triggered, the workflow executes several SQL requests in parallel using AWS Athena. This is achieved by using the “Parallel” state of Step Functions to launch multiple tasks simultaneously.

c. Renaming output files After running the queries, the output files stored in S3 need to be renamed. A Lambda function can be used for this task.

d. Generate a zip with the previously renamed files and send the result to a third-party service such as Slack or email. Once the files have been renamed, they are compressed into a single zip file and sent via a third-party service such as Slack or email. Another Lambda function can be used for this task.

Tips

By hand, not json

To create your POC or your v0, I advise you to do it by hand instead of trying to create your worfklow in json because it’s really not practical and not always very clear, but configuration in json mode is still very useful 😉

Terraform with json

The json is very useful because it will allow you to terraform our work and respect the IAC (Infrastructure As Code).

If you’d like to find out more, I suggest you read this documentation.

Benefits

In this section you will find a list of the advantages of using this service:

1. Design Simplicity: You can design workflows using visual models rather than writing code to manage process steps.

2. Error management: Step Functions manages errors transparently, allowing possible failures to be managed robustly and recovery in the event of a problem.

3. Reliability: Workflows created with Step Functions are durable, without risk of loss of state or duplication of actions.

4. Native integration with AWS services: Step Functions integrates seamlessly with a wide variety of AWS services, allowing you to easily combine different features and services to build flexible and powerful applications.

5. Scalability: The service automatically handles scalability, so you don’t have to worry about capacity constraints.

6. Tracking and Monitoring: Step Functions provides detailed metrics and logs for each step of the workflow, making it easy to track and monitor performance and execution.

7. No maintenance required, as there are no dependencies (unless you use lambdas, of course).

Disadvantages

A contrario here are some negative points of this service.

1. Complicated Review: the json representing your step function can be cumbersome to review in a Pull Request, you will almost always have to connect to the AWS console to study the step function.

2. Costs: Usage-based pricing: AWS Step Functions follows a consumption-based pricing model, which means you will pay based on the number of state transitions, execution time, and events triggered by your workflows. If your workflows are very complex and execute many transitions or process a large number of events, this can result in substantial costs.

Cost Complexity: With Step Functions, it can be difficult to accurately predict costs before using it because they depend on many factors, such as frequency of use, size of data processed, and duration of execution of the workflows. This complexity can make budget management more difficult, especially if your workflows experience unanticipated variations in workload.

3. AWS Dependencies: With a service like Step Functions, it’s true that you could become even more dependent on AWS. However, I consider this to be a false argument, because if you are considering using Step Functions, it is probably because your infrastructure is already tightly integrated with AWS. The idea here is not to promote additional dependency, but rather to leverage the benefits of AWS to make your workflows easier to manage and orchestrate. Indeed, once you’re in the AWS ecosystem, it makes sense to explore services like Step Functions, which can improve operational efficiency, facilitate development, and enable better management of complex workflows.

4. Services that are not fully integrated. As long as you’re using the main AWS services everything’s fine, but as soon as you start using more nested services like AWS Transfer you end up with poorly integrated actions. With the service mentioned above I found myself in the situation where the action returned a success in all cases, even when it didn’t work. For HTTP calls within step function I found myself with unmanaged HTTP headers.

5. Non-encryption of data between steps. As we saw earlier, the steps communicate with each other using json inputs and outputs. None of this data is encrypted, which is fine for most needs, but when we try to set up a rotation of IAM access keys with step functions, we find ourselves in a situation where the AKs are exposed in clear text and as the execution history can be viewed, they are very easy to find!

6. The impossibility of cross-region marketing

Conclusion

This article is coming to an end, although this service is one of my favourites and I think it’s still not used enough, I know that it’s not sometimes. It still needs a few improvements, as you can see.

I hope I’ve given you some ideas and that you’ll give this service a try!