2024-06-10

spend 5 hours trying to figure out a random OOM isssue for llama3, ignoring that the input length varies and a long input at random point will cause OOM