Person:

Malan, David

We introduce CS50 Sandbox, an environment for secure execution of untrusted code. Implemented as an asynchronous HTTP server, CS50 Sandbox offers clients the ability to execute programs (both interactive and non-interactive) written in any compiled or interpreted language in a tightly controlled, resource-constrained environment. CS50 Sandbox’s HTTP-based API takes files, command lines, and standard input as inputs and returns standard output and error plus exit codes as outputs. Atop CS50 Sandbox, we have built CS50 Run, a web- based code editor that enables students to write code in a browser in any language, whether compiled or interpreted, that’s executed server-side within a sandboxed environment. And we have built CS50 Check, an autograding framework that supports black- and white-box testing of students’ code, leveraging CS50 Sandbox to run series of checks against students’ programs, no matter the language of implementation. We present in this work the pedagogical motivations for each of these tools, along with the underlying designs thereof. Each is available as open source.

We propose a client-side virtual machine (VM) as an alternative to on-campus clusters and off-campus clouds as a development environment for students in introductory courses. In Fall 2011, we deployed the CS50 Appliance, our own such VM, to 600 students on campus and, in Fall 2012, to 700 students on campus and 140,000 students online. We present in this work the results of that two-year experiment. The appliance itself is available as open source for others to adapt or adopt. Not only did the appliance enable us to provide students with simpler tools, among them a graphical editor without any latency, it also enabled us to provide more sophisticated tools too, including a web server and database server. Moreover, the appliance ensured that the course's workload no longer required constant Internet access, particularly of students abroad. And the appliance alleviated load on the course's servers, with execution of students' programs now distributed across students' own CPUs. Without the appliance (or more costly clusters or clouds), we certainly could not have accommodated as many as 140,000 students. But some students' laptops, particularly netbooks, struggled under the appliance's weight. Even though designed to be lean, the appliance, like any VM, still consumes resources, particularly RAM. And unforeseen technical difficulties arose in both years, most, but not all, of which we redressed with mid-semester updates and documentation. Overall we have judged our deployment of an appliance a success, superior to past years' clusters and clouds. And we continue to refine the appliance for Fall 2013.

CS50 is Harvard’s introductory course for majors and non-majors alike. For years, we have posted videos of the course’s lectures and sections online for the sake of review and distance education alike. But students’ experience with these videos has been historically passive. Students have been able to watch the course’s content on demand, rewinding and fast-forwarding at will, but they have not had means to engage interactively with the content or to check their understanding of material while watching videos. Furthermore, while we collected basic usage data (e.g., how many times a video was viewed), we lacked detailed analytics describing, for example, which portions of a video were commonly skipped or watched multiple times by students. To make videos more immersive and engaging for students, we developed CS50 Video, an open-source video player for desktop and mobile devices. CS50 Video allows instructors to integrate assessment questions to be answered by students at their own pace or at specific points in time directly into a video player. CS50 Video also allows students to search over video transcripts to find content easily as well as view videos at variable playback speeds (in order to make videos more accessible for ESL learners). Finally, CS50 Video integrates with third-party analytics solutions to allow instructors to view detailed usage statistics describing how students are interacting with videos (e.g., which videos or portions of videos are commonly watched or skipped over). We have deployed CS50 Video to students taking CS50 online and have obtained preliminary results. Because CS50 Video stores responses to questions server-side, we have been able to track students’ performance on in-video assessments. Thus far, we have observed that only 28% of students who watch online videos have engaged with assessment questions. Students who answer an assessment question incorrectly on their first attempt will often try again until reaching a correct answer, with 84.5% of correct answers reached in at most three attempts. We next plan to analyze the effects of in-video assessments on students’ mastery of material and introduce A/B-testing functionality for questions. We also plan to use students’ performance on assessments to understand the topics with which students struggle.

CS50 is Harvard University's introductory course aimed at majors and non-majors alike. Each week, students complete programming assignments and have traditionally received feedback from staff in the form of comments on PDFs of their code. Staff have historically reported spending significant amounts of time grading because of bottlenecks that included generating PDF documents and manually emailing feedback to students. Because we preferred that staff spend less of their time on grading logistics and more time providing feedback and helping students online or in person, we set out to improve the efficiency of the grading process. In Fall 2012, we developed and deployed CS50 Submit, a web-based utility through which staff can leave feedback for students via inline "sticky notes." Following the introduction of CS50 Submit, staff reported grading for 10% fewer hours (i.e., 42 minutes) per week and 13% fewer minutes (i.e., 4 minutes) per student, even while providing as much or more feedback. Meanwhile, we observed significantly higher levels of engagement with the course's online discussion board among staff, suggesting a more favorable distribution of staff workload. With CS50 Submit, we have also been able to audit exactly how much time staff spent grading each week in order to identify additional bottlenecks. Using CS50 Submit, we also observed that, on average, 9% of students each week never read their graders' comments, with a peak one week of 14%. The number of students who did not read feedback increased with time, which has led us to question whether asynchronous, textual comments are the most effective feedback mechanisms for students. In future terms, we plan to experiment with in-person, interactive means of delivering feedback to students. In this paper, we present CS50 Submit and the insights it has yielded into the behavior of students and staff alike.

In Fall 2008, we moved Harvard College’s introductory computer science course, CS50, into the cloud. Rather than continue to rely on our own instructional computing infrastructure on campus, we created a load-balanced cluster of virtual machines (VMs) for our 330 students within Amazon Elastic Compute Cloud (EC2). Our goals were both technical and pedagogical. As computer scientists, we wanted more control over our course’s infrastructure (e.g., root access), so that we ourselves could install software at will and respond to students’ needs at any hour without an IT department between us and our systems. As teachers, we wanted easier access to our students’ work (as via su) as well as the ability to grow and shrink our infrastructure as problem sets’ computational requirements demanded. But we also wanted to integrate into the course’s own syllabus discussion of scalability, virtualization, multi-core processing, and cloud computing itself. What better way to teach topics like those than to have students actually experience them. Although Amazon supported our experiment financially with credits, it was not without costs. Serving as our own system administers cost us time, as did some self-induced late-night technical difficulties. But the upsides proved worth it, as we accomplished our goals. We present in this paper what we did right, what we did wrong, and how we did both so that others can more easily build their own home in the cloud.

Botnets allow adversaries to wage attacks on unprecedented scales at unprecedented rates, motivation for which is no longer just malice but profits instead. The longer botnets go undetected, the higher those profits. I present in this thesis an architecture that leverages collaborative networks of peers in order to detect bots across the same. Not only is this architecture both automated and rapid, it is also high in true positives and low in false positives. Moreover, it accepts as realities insecurities in today’s systems, tolerating bugs, complexity, monocultures, and interconnectivity alike. This architecture embodies my own definition of anomalous behavior: I say a system’s behavior is anomalous if it correlates all too well with other networked, but otherwise independent, systems’ behavior. I provide empirical validation that collaborative detection of bots can indeed work. I validate my ideas in both simulation and the wild. Through simulations with traces of 9 variants of worms and 25 non-worms, I find that two peers, upon exchanging summaries of system calls recently executed, can decide that they are, more likely than not, both executing the same worm as often as 97% of the time. I deploy an actual prototype of my architecture to a network of 29 systems with which I monitor and analyze 10,776 processes, inclusive of 511 unique non-worms (873 if unique versions constitute unique non-worms). Using that data, I expose the utility of temporal consistency (similarity over time in worms’ and non-worms’ invocations of system calls) in collaborative detection. I identify properties with which to distinguish non-worms from worms 99% of the time. I find that a collaborative network, using patterns of system calls and simple heuristics, can detect worms running on multiple hosts. And I find that collaboration among peers significantly reduces the risk of false positives because of the unlikely, simultaneous appearance across peers of non-worm processes with worm-like properties.

In recent months [teachers have] become publishers of content and students subscribers thereof by way of podcasts, feeds of audio, video, and other content that can be downloaded to clients like iTunes and devices like iPods. In the fall of 2005, we ourselves began to podcast Harvard Extension School's Computer Science E-1 in both audio and video formats, the first course within Harvard University to do so. Our goals were to provide students with more portable access to educational content and to involve them in technology itself.To evaluate this experiment, we have analyzed logs and surveys of students. We find that our students valued E-1's podcast more as a vehicle for review (45%) than as an alternative to attendance (18%). We also find that most students (71%) tended to listen to or watch lectures on their computers, with far fewer relying upon audio-only (19%) or video (10%) iPods. We argue, meanwhile, that podcasting, despite its widespread popularity, is but a marginal improvement on trends long in progress. It is this technology's reach that we claim is significant, not the technology itself. Logs suggest that E-1's own podcast, available not only to students but to the public at large, has acquired (as of September 2006) between 6,000 and 10,000 subscribers from over 50 countries. We argue, then, that podcasting offers to extend universities' educational reach more than it offers to improve education itself.

We present the first known implementation of elliptic curve cryptography over F2p for sensor networks based on the 8-bit, 7.3828-MHz MICA2 mote. Through instrumentation of UC Berkeley's TinySec module, we argue that, although secret-key cryptography has been tractable in this domain for some time, there has remained a need for an efficient, secure mechanism for distribution of secret keys among nodes. Although public-key infrastructure has been thought impractical, we argue, through analysis of our own implementation for TinyOS of multiplication of points on elliptic curves, that public-key infrastructure is, in fact, viable for TinySec keys' distribution, even on the MICA2. We demonstrate that public keys can be generated within 34 seconds, and that shared secrets can be distributed among nodes in a sensor network within the same, using just over 1 kilobyte of SRAM and 34 kilobytes of ROM.

In Fall 2007, we introduced “virtual office hours” into Harvard College’s introductory computer science course, CS 50, so that students could meet with teaching fellows (TFs) online to discuss problem sets at any hour from anywhere. Our goals were to lower the bar to interaction among TFs and students and to improve the efficiency and convenience of the same. Rather than rely on email and online forums alone, we experimented with Elluminate, third-party software that not only allowed students and TFs to chat via IM and VOIP, it also enabled the latter to see and even share control of the former’s screens (e.g., code in students’ terminal windows). Students, in turn, were able to troubleshoot bugs with TFs by their (virtual) side. We surveyed our nearly 300 students on their experiences with office hours, both physical and virtual. Although most students responded positively to the idea of virtual office hours, only 55% logged in at least once. However, nearly the same number (62%) attended the physical. We ultimately judged our virtual office hours a net positive, with 14% of students attending the virtual (and 21% the physical) “often.” But our experiment was not without some unexpected results. We found that wait times online sometimes matched or exceeded those in the physical lab, partly the result of the software’s own shortcomings and students’ habits online. Ultimately, the audience for these virtual office hours was entirely self-selecting. Those students who liked the experience online opted in, whereas those who preferred more traditional help opted out.

We present a critical evaluation of the first known implementation of elliptic curve cryptography over F_2p for sensor networks based on the 8-bit, 7.3828-MHz MICA2 mote. We offer, along the way, a primer for those interested in the field of cryptography for sensor networks. We discuss, in particular, the decisions underlying our design and alternatives thereto. And we elaborate on the methodologies underlying our evaluation.

Through instrumentation of UC Berkeley's TinySec module, we argue that, although symmetric cryptography has been tractable in this domain for some time, there has remained a need, unfulfilled until recently, for an efficient, secure mechanism for distribution of secret keys among nodes. Although public-key infrastructure has been thought impractical, we show, through analysis of our original implementation for TinyOS of point multiplication on elliptic curves, that public-key infrastructure is indeed viable for TinySec keys' distribution, even on the MICA2. We demonstrate that public keys can be generated within 34 seconds and that shared secrets can be distributed among nodes in a sensor network within the same time, using just over 1 kilobyte of SRAM and 34 kilobytes of ROM. We demonstrate that communication costs are minimal, with only 2 packets required for transmission of a public key among nodes. We make available all of our source code for other researchers to download and use. And we discuss recent results based on our work that corroborate and improve upon our conclusions.