Cloud Computing►AWS

What to Do When You Hit AWS Lambda's 6MB Payload Limit

Introduction

AWS Lambda is a powerful serverless compute service, but it comes with some constraints, including a 6MB payload limit for synchronous invocations. If your application needs to handle larger payloads, you might wonder how to address this limitation effectively. In this post, we’ll explore various strategies to overcome the 6MB limit and keep your architecture scalable and efficient.

1. Use S3 for Large Payloads

Offload large files or data to Amazon S3 and pass the S3 object key as an input to your Lambda function. This approach keeps the payload small and makes it easier to manage large datasets.

Workflow Example:

  1. Upload the large payload to an S3 bucket.
  2. Trigger the Lambda function with the S3 object key as input.
  3. Fetch and process the object inside the Lambda function.

2. API Gateway Service Proxy

API Gateway can act as a service proxy to directly integrate with AWS services like S3. This approach avoids Lambda altogether and leverages API Gateway’s 10MB payload limit.

Benefits:

  • Offloads processing from Lambda.
  • Simplifies architecture by directly interacting with S3 or other AWS services.

3. Use Presigned S3 URLs

Generate presigned URLs to allow clients to upload large files directly to S3. This eliminates the need to send large payloads to Lambda or API Gateway.

Steps:

  1. Generate a presigned URL from your backend application.
  2. Share the URL with the client.
  3. The client uploads the file directly to S3.

4. Lambda@Edge for Forwarding

Use Lambda@Edge with a CloudFront distribution to handle authentication and route requests directly to S3. This method provides a simple HTTP endpoint for clients but adds some operational complexity.

Use Case:

  • Ideal for applications requiring secure, low-latency access to large files.

5. Presigned POST

Similar to presigned URLs, this method uses a POST policy to restrict file content type, size, and metadata. Clients upload files to S3 via HTTP POST, adhering to the provided restrictions.

Key Features:

  • Greater control over uploads.
  • Enables additional constraints like allowed file types.

Choosing the Right Approach

Each solution has trade-offs in complexity, scalability, and client-side implementation. Heres how to choose:

  • Use S3 and presigned URLs for simplicity and scalability.
  • Use API Gateway service proxy for integrating directly with AWS services.
  • Use Lambda@Edge for applications requiring low latency or custom logic at the edge.

Conclusion

While the 6MB payload limit can be a constraint, AWS provides several tools and services to work around it effectively. By leveraging S3, presigned URLs, or API Gateway, you can handle large payloads without compromising scalability or performance.