Software Engineering Researcher

Jon Bell

About CV

About Jon

Jon is a fourth year PhD candidate at Columbia University studying Software Engineering with Prof Gail Kaiser. His research interests in Software Engineering mostly fall under the umbrella of Software Testing and Program Analysis. Jon’s recent research in accelerating software testing has been recognized with an ACM SIGSOFT Distinguished Paper Award (ICSE ’14 – Unit Test Virtualization with VMVM), and has been the basis for an industrial collaboration with the bay-area software build acceleration company Electric Cloud. Jon actively participates in the artifact evaluation program committees of ISSTA and OOPSLA, and has served several years as the Student Volunteer chair for OOPSLA. His most recent publications are Efficient Dependency Detection for Safe Java Test Acceleration (FSE ’15), Pebbles: Fine-Grained Data Management Abstractions for Modern Operating Systems (OSDI ’14) and Phosphor: Illuminating Dynamic Data Flow in Off-The Shelf JVMs (OOPSLA ’14).

A very long time ago, he co-founded RentPost and served as an intern at Codestreet, Sikorsky Aircraft and MediaMerx.

His other interests include photography and cycling.

More information about Jon, including a complete publications listing, is available in his Curriculum Vitae.


Research Overview

Slow builds remain a plague for software developers. The frequency with which code can be built (compiled, tested and packaged) directly impacts the productivity of developers: longer build times mean a longer wait before determining if a change to the application being built was successful. We have discovered that in the case of some languages, such as Java, the majority of build time is spent running tests, where dependencies between individual tests are complicated to discover, making many existing test acceleration techniques unsound to deploy in practice. Without knowledge of which tests are dependent on others, we cannot safely parallelize the execution of the tests, nor can we perform incremental testing (i.e., execute only a subset of an application’s tests for each build). The previous techniques for detecting these dependencies did not scale to large test suites: given a test suite that normally ran in two hours, the best-case running scenario for the previous tool would have taken over 422 CPU days to find dependencies between all test methods (and would not soundly find all dependencies) — on the same project the exhaustive technique (to find all dependencies) would have taken over 1e300 years. We present a novel approach to detecting all dependencies between test cases in large projects that can enable safe exploitation of parallelism and test selection with a modest analysis cost. Read more about our tool, ElectricTest in our ESEC/FSE 2015 paper.

Testing large software packages can become very time intensive. To address this problem, researchers have investigated techniques such as Test Suite Minimization. Test Suite Minimization reduces the number of tests in a suite by removing tests that appear redundant, at the risk of a reduction in fault-finding ability since it can be difficult to identify which tests are truly redundant. We take a completely different approach to solving the same problem of long running test suites by instead reducing the time needed to execute each test, an approach that we call Unit Test Virtualization. With Unit Test Virtualization, we reduce the overhead of isolating each unit test with a lightweight virtualization container. We describe the empirical analysis that grounds our approach and provide an implementation of Unit Test Virtualization targeting Java applications. We evaluated our implementation, VMVM, using 20 real-world Java applications and found that it reduces test suite execution time by up to 97% (on average, 62%) when compared to traditional unit test execution. We also compared VMVM to a well known Test Suite Minimization technique, finding the reduction provided by VMVM to be four times greater, while still executing every test with no loss of fault-finding ability. For more information about VMVM, please see our paper, and github project.

Dynamic taint analysis is a well-known information flow analysis problem with many possible applications. Taint tracking allows for analysis of application data flow by assigning labels to data, and then propagating those labels through data flow. Taint tracking systems traditionally compromise among performance, precision, soundness, and portability. Performance can be critical, as these systems are often intended to be deployed to production environments, and hence must have low overhead. To be deployed in security-conscious settings, taint tracking must also be sound and precise. Dynamic taint tracking must be portable in order to be easily deployed and adopted for real world purposes, without requiring recompilation of the operating system or language interpreter, and without requiring access to application source code.

We present Phosphor, a dynamic taint tracking system for the Java Virtual Machine (JVM) that simultaneously achieves our goals of performance, soundness, precision, and portability. Moreover, to our knowledge, it is the first portable general purpose taint tracking system for the JVM. We evaluated Phosphor’s performance on two commonly used JVM languages (Java and Scala), on two successive revisions of two commonly used JVMs (Oracle’s HotSpot and OpenJDK’s IcedTea) and on Android’s Dalvik Virtual Machine, finding its performance to be impressive: as low as 3% (53% on average; 220% at worst), using the DaCapo macro benchmark suite. Our OOPSLA 2014 paper describes our approach toward achieving portable taint tracking in the JVM, which is released under an MIT license via GitHub.

As software continues to grow in complexity, bugs remaining after deployment have become increasingly challenging to resolve. Resolving such bugs begins with reproduction, which is time consuming and difficult when the “root cause” involves nondeterministic factors such as timing and interleaving — insidious on multicore hardware. Because conventional bug reports are inadequate for reproduction of tricky bugs, deterministic record-replay has been developed to capture bugs in the field and replay them in the lab, but most previous approaches require special hardware or add very high overhead. More concerning, these approaches typically transmit too much information back to developers, including sensitive information in bug reports. We seek to create systems to efficiently reproduce field failures while maintaining user privacy. For information on early efforts in this direction, please see Chronicler.

More Jon Bell