![]() ![]() Execute an S3 GET request for a given bucket and key.Here is the basic structure of the Lambda function: We used the excellent cargo-lambda tool to generate a Rust project for our Lambda function and compile the release binary. In the rest of the blog post, we will cover what the code looked like for each language, how performance varied as we tried different CPU architectures and JSON parsing libraries for each language, how cold start times differed between languages, and how memory allocation affected S3 download speed. At Scanner, we use Mozilla’s bincode library in our index file format, giving us a 4x performance boost over SIMD-accelerated JSON parsing.There are data formats that are much faster than JSON.Under the hood, AWS Lambda probably runs your function on a bigger machine with a better network adapter. Even if your program does not need very much memory, consider allocating 640MB of memory or more to your Lambda function if you need to maximize S3 download throughput.You reach maximum S3 read throughput (~90MB/sec) when you allocate 640MB of memory or more.In fact, our Go-based function using the standard library parser was slower than our Python-based function, but our Go function using the fast parser was insanely fast. In Go, the standard library’s JSON parser in encoding/json was 10x slower than the parser in valyala/fastjson.Pick your data parsing libraries carefully.Performance improved as we allocated more memory until we reached a plateau around 1.5 GB.Increasing memory allocation also increases performance, to a point.That’s 4x faster than Java, and 6x faster than Python. At peak performance, they took only 2 seconds to process 1 GB of logs.At peak performance, they took 8 – 12 seconds to process 1 GB of logs.The lower the task duration, the faster the performance. The chart above shows the performance results for the fastest performing settings for each language (Python, Java, Go, and Rust) at varying memory allocation levels. We ran 10 test runs from a cold start for each choice of language, CPU architecture, JSON parser, and memory allocation.įor each Lambda function invocation, we recorded the cold start time and the total time to process 1GB of JSON logs in S3. Memory allocation: A series of values between the minimum allowed Lambda function memory allocation (128MB) and the maximum (10GB).JSON parser: Try standard library JSON parser, and try the fastest JSON parser we could find for the language.There were 4 parameters for each test run: Given 1 GB of JSON log events in S3, run a Lambda function from a cold start to stream, decompress, and parse the objects on the fly. We compared the languages by giving them a classic, “bursty” data lake task: We pitted the four languages against one another to see which was the fastest, and here is what we found. We use Rust for this use case, but we wanted to know how Rust compared with Go, Java, and Python in terms of performance. At Scanner, we use serverless Lambda functions to perform fast full-text search over large volumes of logs in data lakes, and our queries need to be lightning fast. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |