Can't make CRaC run on AWS Lambda

[Vadym79]

I took azul/zulu-openjdk:21-jdk-crac-latest and deployed it as a Docker mage on Lambda (see https://github.com/Vadym79/AWSLambdaJavaDockerImageWithCrac/tree/master/spring-crac-with-serverless-container oder https://github.com/Vadym79/AWSLambdaJavaDockerImageWithCrac/tree/master/example-lambda)

At the CRaC checkpoint I get the following error:

Detected cgroup V1 freezer

Warn (compel/src/lib/infect.c:129): Unable to interrupt task: 8 (Operation not permitted)

Unlock network

Unfreezing tasks into 1

Unseizing 8 into 1

Error (compel/src/lib/infect.c:358): Unable to detach from 8: Operation not permitted

Error (criu/cr-dump.c:2063): Dumping FAILED.

I have also tried Anton's way from this example (but using Java 21 not Java 8), namely using AWS Lambda Emulator on AWS Cloud9, starting the application there, making checkpoint and packaging the data from it into Docker Image deployed on AWS Lambda in the hope that CRaC Restore will then work (see the sae repo (see https://github.com/Vadym79/AWSLambdaJavaDockerImageWithCrac/tree/master/spring-crac-with-serverless-container). It does not with the following error:

166: Error (criu/cr-restore.c:3151): rseq: can't restore as kernel doesn't support it

Error (criu/cr-restore.c:1548): 166 exited, status=1

Error (criu/cr-restore.c:2605): Restoring FAILED.

Sure, it may be the case that the kernel of the AWS Cloud 9 instance is a different one than on the Firecracker VM where Lambda is running and CRIU is very sensitive to it.

Can you please advice? Did Firecracker VM stop supporting CRIU/CRaC? I got an inofficial comment from AWS that Firecracker VM doesn't support CRaC but couldn't veriy it.

Thanks for your support !

[rvansa]

The dump error looks like you're not having SYS_PTRACE capability, because
a) not running as root (either directly or jdk/lib/criu owner running as root and having suid bit set)
b) in a containerized environment either using --privileged or --cap-add SYS_PTRACE --cap-add CHECKPOINT_RESTORE
c) having wrong value in /proc/sys/kernel/yama/ptrace_scope

I was trying to follow https://github.com/Vadym79/AWSLambdaJavaDockerImageWithCrac/tree/master/example-lambda but image build failed at COPY checkpoint.cmd.sh - the file is not present (I wasn't really looking into why you use a modified version of the scripts).

The restore error about rseq is not important, though it indicates that you either use a very old kernel or have some seccomp policies that will affect feature detection. What AMI is that?

[Vadym79]

Hi Radim,
thanks for your reply. My example is probably difficult to follow as I published the Docker Image in my private Amazon ECR repository, But nevetheless if I run Docker image on AWS Lambda I don't control the docker run command and can't set neither --provilleged nor everything else. But to create the Docker image I used m5 AWS Cloud9 instance which has Amazon Linux 2023 and Docker pre-installed.
But you first can try it on your own in this example-lambda. You currently use Java 1.8 (also in template.yaml) there which isn't supported by AWS Lambda anymore since end of 2023. You can neither deploy nor update Lambda with this runtime. So please update everything to Java 21 and re-test if this still works. You can also skip this very complex example with packaging "local" checkpoint file into Docker image and then doing restore. Simply do checkpoint in the Lambda function itself during its execution first to test whether CraC/CRIU works on AWS Lambda in general.

[rvansa]

But nevetheless if I run Docker image on AWS Lambda I don't control the docker run command and can't set neither --provilleged nor everything else.

Well it's the known current limitation of CRaC using CRIU underhood: you cannot checkpoint without some extra privileges. We know that for some use cases it's inconvenient, and there can be a solution in the future, but not ATM.

But you first can try it on your own in this example-lambda.

That's why I tried to follow your linked example locally, but there was the failure. Can you fix it up so that the provided steps work?

You currently use Java 1.8

Err, why do you think that? When trying to follow the tutorial I've executed docker run -it --rm azul/zulu-openjdk:21-jdk-crac (the image you mentioned in the beginning) in one terminal and then copied /opt/zulu21.34.19-ca-crac-jdk21.0.3-linux_x64 and provided this to jlink.

[Vadym79]

"Well it's the known current limitation of CRaC using CRIU underhood: you cannot checkpoint without some extra privileges. We know that for some use cases it's inconvenient, and there can be a solution in the future, but not ATM."
->
yes I suppose that this example was able to run on AWS Lambda with Java 8 several years ago with exactly the same limitations. But I couldn't make it work with Java 21.

I meant that your own example https://github.com/CRaC/example-lambda uses Java 8 (see https://github.com/CRaC/example-lambda/blob/master/template.yml with Runtime: java8) and pom.xml. So this is not a valid tutorial as AWS Lambda doesn't support Java 8 anymore. That's why I suggested you to update your own example to Java 21 and retest whether it still supposed to work or something has change on the Firecracker VM.

[rvansa]

I see what you meant - it's a bit confusing as the example is based on another example, but AFAIU the template.yml is not used in the CRaCed version. We should clean it up of non-relevant parts. By following the tutorial I meant the steps as provided in the README

[rvansa]

@Vadym79 I've filed #8 with removal of those unneeded files & some convenience changes. I've verified that this works locally; can you check where you diverge from the steps or at what phase does the example stop working?

[Vadym79]

@rvansa Thanks, do you mean it works locally with AWS Lambda emulator or what I tested on AWS Lambda itself (this is where I get the error

166: Error (criu/cr-restore.c:3151): rseq: can't restore as kernel doesn't support it

Error (criu/cr-restore.c:1548): 166 exited, status=1

Error (criu/cr-restore.c:2605): Restoring FAILED.

?

[rvansa]

Now I've tested in AWS as well; created function crac-test from the container image, and in the Test tab use the SQS template:

It spits out Error (criu/cr-restore.c:2009): Can't attach to 129: Operation not permitted in the log but this doesn't seem to prevent it from working. I'll check CRIU sources why is this an error...

[Vadym79]

@rvansa Thanks, ok, it seems generally to work , which is good. Can you please use this example https://github.com/Vadym79/AWSLambdaJavaDockerImageWithCrac/tree/master/spring-crac-with-serverless-container and https://github.com/Vadym79/AWSLambdaJavaDockerImageWithCrac/blob/master/spring-crac-with-serverless-container/crac-steps.sh. I worked on this several months ago, but I had to modify some scripts, but I couldn't make it work on AWS Cloud 9 instance the way it was described in your example. Maybe I missed something important there.

[rvansa]

@Vadym79 I need a bit more detailed steps to follow to reproduce this; the readme looks like a mix of local setup and outdated information.

I've started a fresh Cloud9, cloned the repository and fetched latest JDK 22 with CRaC, ensuring that criu is owned by root and has SUID set. Installed Maven. Started local DynamoDB.
I ran ./crac-steps.sh s00_init and s01_build_aws. The s02_start_checkpoint does not use the image with aws, and when I just replaced the image in the docker command but that fails with Could not get environment variable AWS_LAMBDA_RUNTIME_API

So another attempt, I've built the image using s01_build and started using s02_start_checkpoint, executing ./crac-steps.sh s03_checkpoint in another terminal, local checkpoint succeeded:

CR: Checkpoint ...
END RequestId: 5c9a9c0e-502f-4511-a24e-f5030de5e075
REPORT RequestId: 5c9a9c0e-502f-4511-a24e-f5030de5e075  Init Duration: 0.60 ms  Duration: 19632.44 ms   Billed Duration: 19633 ms       Memory Size: 3008 MB    Max Memory Used: 3008 MB
10 Jul 2024 08:06:32,621 [WARNING] (rapid) First fatal error stored in appctx: Runtime.ExitError
10 Jul 2024 08:06:32,621 [WARNING] (rapid) Process 165(bash) exited: Runtime exited without providing a reason

Running ./crac-steps.sh s04_prepare_restore but a local deploy fails because of

Suppressed: java.nio.file.NoSuchFileException: /tmp/crac/dump4.log
                at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
                at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:261) ~[?:?]
                at java.base/java.nio.file.Files.newByteChannel(Files.java:379) ~[?:?]
                at java.base/java.nio.file.Files.newByteChannel(Files.java:431) ~[?:?]
                at java.base/java.nio.file.Files.readAllBytes(Files.java:3268) ~[?:?]
                at software.amazonaws.PrimingResource.afterRestore(PrimingResource.java:49) ~[function/:?]

I've added RUN mkdir -p /tmp/crac && touch /tmp/crac/dump4.log to the restore Dockerfile and tried again. This time it seems to work:

[main] INFO software.amazonaws.PrimingResource - 
2024-07-10 08:16:55  INFO  DefaultLifecycleProcessor:576 - Restarting Spring-managed lifecycle beans after JVM restore
2024-07-10 08:16:55  INFO  AWSLambda:56 - Restored AWSLambda in 0.157 seconds (process running for 0.164)
[main] INFO com.amazonaws.serverless.proxy.internal.servlet.AwsServletContext - Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-07-10 08:16:55  INFO  DispatcherServlet:532 - Initializing Servlet 'dispatcherServlet'
2024-07-10 08:16:55  INFO  DispatcherServlet:554 - Completed initialization in 2 ms
[main] INFO software.amazonaws.example.product.handler.StreamLambdaHandler - entered generic stream lambda handler
[main] INFO software.amazonaws.example.product.controller.ProductController - entered getProductById method with id  crac0
[main] INFO software.amazonaws.example.product.dao.DynamoProductDao - product table name AWSLambdaSpringBoot32Java21DockerImageAndCRaCProductsTable
[main] INFO software.amazonaws.example.product.dao.DynamoProductDao - endpoint http://172.17.0.1:8000
[main] INFO software.amazonaws.example.product.dao.DynamoProductDao - end get item 
[main] INFO software.amazonaws.example.product.controller.ProductController -  product not found 
[main] INFO com.amazonaws.serverless.proxy.internal.LambdaContainerHandler - null null- null [10/07/2024:08:16:56Z] "GET /products/0 null" 200 4 "-" "-" combined
END RequestId: 9b618ec0-c4fc-4619-b10e-6099c13d28cd
REPORT RequestId: 9b618ec0-c4fc-4619-b10e-6099c13d28cd  Init Duration: 0.64 ms  Duration: 1148.88 ms    Billed Duration: 1149 ms        Memory Size: 3008 MB    Max Memory Used: 3008 MB

So in what step exactly is the problem?

[Vadym79]

@rvansa Thanks! Do I understand correctly that the last console output is on AWS Lambda and not locally?

I personally had

166: Error (criu/cr-restore.c:3151): rseq: can't restore as kernel doesn't support it

Error (criu/cr-restore.c:1548): 166 exited, status=1

Error (criu/cr-restore.c:2605): Restoring FAILED.

during the CraC restore on AWS Lambda. But maybe I missed several steps. For example your wrote "ensuring that criu is owned by root and has SUID set." I didn't do that Can you please give me exact commands to execute?

Also what Docker image have you used for the latest JDK 22 with CRaC?

[rvansa]

during the CraC restore on AWS Lambda. But maybe I missed several steps. For example your wrote "ensuring that criu is owned by root and has SUID set." I didn't do that Can you please give me exact commands to execute?

sudo chown root $JAVA_HOME/lib/criu && sudo chmod u+s $JAVA_HOME/lib/criu

Most CRaC instructions suggest sudo tar ... when unpacking the JDK but I personally prefer to assert it this way.

Also what Docker image have you used for the latest JDK 22 with CRaC?

This question does not make any sense. The image is selected by the Docker scripts that are part of the repo. I've just added Zulu 22.30.13 from https://www.azul.com/downloads

[rvansa]

Thanks! Do I understand correctly that the last console output is on AWS Lambda and not locally?

No, your repository does not have s06_init_aws and later steps, contrary to https://github.com/CRaC/example-lambda

166: Error (criu/cr-restore.c:3151): rseq: can't restore as kernel doesn't support it

I should have reacted to this part earlier. This looks like you've did the checkpoint in a container with GLIBC >= 2.35 and modern kernel (not exactly sure which version but Amazon offers you the AMI 2023 with 6.something = modern). If you later try to restore with AMI 2 that features kernel 5.10 you run into this trouble. The solution is to either base this on an older image (e.g. Ubuntu 18, or CentOS 7) that does not use rseq, or perform the checkpoint with environment variable GLIBC_TUNABLES set to glibc.pthread.rseq=0.

[Vadym79]

Ok, it worked for me locally to, but not on AWS Lambda. Yes, I see that there are indeed no instructions how to run it on AWS, but basically there are 2 steps

1. Push the Docker image containing Checkpoint file to Amazon ECR and give it the name aws-spring-boot-3.2-java21-crac-custom-docker-image:v1 to execute 2) without any changes
2. execute sam deploy -g (Cloud 9 has SAM installed). This docker image is referenced in the template.yaml, see https://github.com/Vadym79/AWSLambdaJavaDockerImageWithCrac/blob/master/spring-crac-with-serverless-container/template.yaml

I will also look later and add the exact instructions.

[Vadym79]

See my message above

Here are instructions for 1) I use eu-central-1 AWS region. Please replace {aws_account_id} with your aws account id.

a) aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin {aws_account_id}.dkr.ecr.eu-central-1.amazonaws.com

b) in case the reporsitory doesn't exist

aws ecr create-repository --repository-name aws-spring-boot-3.2-java21-crac-custom-docker-image --image-scanning-configuration scanOnPush=true --region eu-central-1

c) docker tag crac-lambda-restore-zulu-spring-boot {aws_account_id}.dkr.ecr.eu-central-1.amazonaws.com/aws-spring-boot-3.2-java21-crac-custom-docker-image:v1

docker push {aws_account_id}.dkr.ecr.eu-central-1.amazonaws.com/aws-spring-boot-3.2-java21-crac-custom-docker-image:v1

Then deploy the stack with AWS SAM: sam deploy -g