CS/SWE 795, Fall 2017, Homework 2, due 9/18/2017 4:00pm.
Introduction
This week, we will take a look at Phosphor, using it in the context of testing. Specifically, we will implement a simple version of the OraclePolish tool by Chen Huo and James Clause. The idea is that a test case might have brittle assertions, which depend on values not specified by the test, or be over-specified, where the test case provides inputs to the system under test, but never checks values derived from those inputs. A brittle assertion might break on future versions of the system we are testing, because we are creating assertions that are checking things that we are not purposely providing as inputs. A test that is over-specified might break in future versions of the system under test because we are providing inputs to it that are not necessarily used currently by the system under test, but in the future, maybe they will use in some other way.
OraclePolish detects these brittle assertions and unused inputs using taint tracking (with implicit flow tracking), and is built on JPF (which we’ll talk about next week). For your assignment this week, you will see how to implement a technique for detecting unused inputs in tests using Phosphor. Phosphor is orders of magnitude faster than JPF, and is compatible with a full range of Java applications (whereas JPF is a model checking tool, and is not designed to be used with large applications).
For simplicity, you will only consider brittle assertions. The implementation strategy is relatively straightforward: whenever a constant value (e.g. string, integer, float, double, long, etc) is created in a test (e.g. by a LDC, ICONST_N, DCONST_N, BIPUSH, SIPUSH instruction), we want to taint that value. At each assertion, we want to check the taint mark of the value we are assert’ing on. If it’s not tainted, then it’s a value that was not loaded in by the test code.
Getting Started
This assignment totals 100 points, and will correspond to 10% of your final grade for this course.
Start out by importing the HW2 GitHub repository. Then, clone your private repository on your machine, and import it to your IDE. I’ve provided a starter project that uses Phosphor, configured with implicit flow tracking. You will do all of your work for this assignment in this project.
Marking inputs to tests as tainted (50 points)
For simplicity, we will consider only the inputs in test files themselves as “controlled” inputs. The starter code provides a method visitor for you to build in that is already hooked in to the entire Phosphor process, edu.gmu.cs795.hw2.inst.HW2MethodVisitor.
To mark inputs as tainted, we want to transform bytecode so that when an input is loaded, that input is tainted. You should consider the following bytecode instructions as “providing inputs,” when they occur in test code (that is, in a class file ending in the name ‘IT’):
- BIPUSH
- SIPUSH
- LDC
- ICONST_*
- LCONST_*
- FCONST_*
- DCONST_*
Implement your method visitor so that right after each of the instructions, there is a call to a runtime method that you create in edu.gmu.cs795.hw2.runtime.TaintRuntime. Recall that Phosphor does crazy stuff to store taint tags for primitive values, creating extra arguments for your methods and magically passing those values around. As an example, here is code that you might use to generate a call to tag an integer:
|
1 |
super.visitMethodInsn(INVOKESTATIC, Type.getInternalName(TaintRuntime.class), "ldc", "(I)I", false);
|
Then, on TaintRuntime, you would create the method:
|
1
2
3
4
5
6
|
public static TaintedIntWithObjTag ldc$$PHOSPHORTAGGED(Taint tag, int in, TaintedIntWithObjTag ret)
{
ret.taint = new Taint("Some label");
ret.val = in;
return ret;
}
|
It does not matter what you make the taint mark – as long as it is non-null.
You only need to consider primitive types (e.g. not objects or arrays or strings).
Checking taint marks on assertions (50 points)
JUnit provides a variety of assert* methods in the class org.junit.Assert, which are used to check various values produced by your test. We want to check the arguments passed to these methods. The easiest way to do this is to modify the implementation of each assert* method, for instance:
|
1
2
3
|
static public void assertTrue(boolean condition) {
assertTrue(null, condition);
}
|
to check the taint tag on each value being asserted. In Java source code, this might look like
|
1
2
3
4
|
static public void assertTrue(boolean condition) {
TaintRuntime.checkTaintTagIsNonNull(condition);
assertTrue(null, condition);
}
|
, where TaintRuntime.checkTaintTagIsNonNull will use Phosphor to check the tag on the condition, and throw an edu.gmu.cs795.hw2.runtime.BrittleAssertionException.
You should programmatically (e.g. in your method visitor) rewrite each assertEquals and assertTrue method in org.junit.Assert to do this check. You only need to consider the assertEquals methods for primitive types (plus assertTrue).
Testing notes
You can run the whole thing by typing mvn verify. For your convenience, I’ve provided a few sample tests to get you started: one that is brittle, and one that is not. JUnit will complain when you run mvn verify (without starting to implement this assignment) because it is expecting you to throw an exception in the brittle test.
Submission
Perform all of your work in your homework-2 git repository. Commit and push your assignment. Make sure that your name is specified somewhere in the README. The time that you push the code will be the time used to determine if it is on time, within the 24-hour grace period, or too late.
Make sure that your released code includes all of your files and builds properly. You can do this by clicking to download the archive, and inspecting/trying to build it on your own machine/vm. There is no need to submit binaries/jar files (the target/ directory is purposely ignored from git).
If you want to resubmit before the deadline: Simply commit and push again. I will grade the last commit that was received before the deadline. You will not be able to push after the deadline.