converting image into buffer is too slow

[gabrieleolmi]

I did some tests with 10 MB images and noticed that the function transformToByteArray takes 0.5 seconds to 1 second to convert the image stream to a buffer. Sharp supports streams, could we try compressing images without turning them into buffers?

image-optimization/functions/image-processing/index.mjs

Line 36 in 259f680

originalImageBody = getOriginalImageCommandOutput.Body.transformToByteArray();

[achrafsouk]

Thank you, we will investigate !

[piotrekwitkowski]

We have an initial version with streams working internally and are now collecting data on performance of streams vs conversion to byte array. In the meantime, @gabrieleolmi, could you tell us about your setup, such as:

• what format of photos did you use for input and output,
• what was the input and output resolution,
• is the time (up to 1 second) spent on the transformation to byte array a regression from an earlier version of the solution that you used in the past? Did you notice a conversion slowdown at some point, for example after moving to another Lambda runtime version? Any other changes in the code or the CDK setup that we should be aware of, or points of reference for us?

[gabrieleolmi]

I've been using this repository for a short time so I don't know if the speed has gotten worse with the latest versions. In my opinion no, it is precisely the conversion to byte array that is slow. I tried images of various formats (jpg, png, webp) and in high resolution, slightly compressed (5/10/20 MB). The larger the image size, the longer it takes to convert to byte array. I use Node.js version 18

[KevinToala]

Hi 👋

I am experiencing the same problem when processing an 8MB image. I had to increase the response time in cloudfront up to 60 seconds to allow the image to be processed and returned without error but I see that sometimes it even takes longer than 60 seconds.
Do we have maybe some example of how to use streams and not transformToByteArray() ? I was researching but I couldn't find an example for aws lambda

[amiantos]

I've have a 7mb, 7000x5000 JPEG that the lambda function seems to hang on when trying format=auto. However, that image can be successfully resized and converted into other formats just fine, and quickly. This led me to believe the issue isn't due to transformToByteArray, but due to RAM usage when converting very large images between different formats.

I ended up bumping the lambda task RAM up to 4096 (max ram used is only 1238 during this process) and the Lambda timeout to 5 minutes during testing, and converting this image to AVIF took 57.968 seconds. So upping the CF timeout is likely necessary but not ideal.

I think there may be a companion issue where the lambda function cannot return the full image via base64 and you get a JSON error in Firefox when initially loading a large converted image. Maybe there is a payload limit being hit in lambda? I may make a separate issue for this to discuss it if I figure out how to reproduce it consistently.