Virtualizing Office Hours in CS 50
School of Engineering and Applied Sciences Harvard University Cambridge, Massachusetts, USA

David J. Malan

malan@post.harvard.edu

ABSTRACT
In Fall 2007, we introduced “virtual office hours” into Harvard College’s introductory computer science course, CS 50, so that students could meet with teaching fellows (TFs) online to discuss problem sets at any hour from anywhere. Our goals were to lower the bar to interaction among TFs and students and to improve the efficiency and convenience of the same. Rather than rely on email and online forums alone, we experimented with Elluminate, third-party software that not only allowed students and TFs to chat via IM and VOIP, it also enabled the latter to see and even share control of the former’s screens (e.g., code in students’ terminal windows). Students, in turn, were able to troubleshoot bugs with TFs by their (virtual) side. We surveyed our nearly 300 students on their experiences with office hours, both physical and virtual. Although most students responded positively to the idea of virtual office hours, only 55% logged in at least once. However, nearly the same number (62%) attended the physical. We ultimately judged our virtual office hours a net positive, with 14% of students attending the virtual (and 21% the physical) “often.” But our experiment was not without some unexpected results. We found that wait times online sometimes matched or exceeded those in the physical lab, partly the result of the software’s own shortcomings and students’ habits online. Ultimately, the audience for these virtual office hours was entirely self-selecting. Those students who liked the experience online opted in, whereas those who preferred more traditional help opted out.

50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% "none" "a little" "a lot"

Figure 1: When asked via surveys to qualify their prior programming experience, 44% of students responded with “none,” 46% with “a little,” and 10% with “a lot.” Whereas “a little” generally meant a course in high school, “a lot” generally meant even more experience than AP Computer Science (e.g., self-taught).

General Terms
Experimentation, Human Factors

Keywords
collaboration, CSCW, distance education, virtual office hours

1. Categories and Subject Descriptors
K.3.1 [COMPUTERS AND EDUCATION]: Computer Uses in Education—Collaborative learning, Distance learning; K.3.2 [COMPUTERS AND EDUCATION]: Computer and Information Science Education—Computer science education

INTRODUCTION

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ITiCSE’09, July 6–9, 2009, Paris, France. Copyright 2009 ACM 978-1-60558-381-5/09/07 ...$5.00.

Computer Science 50 is Harvard College’s introductory course for majors and non-majors alike, a one-semester amalgam of courses generally known as CS 1 and CS 2 taught mostly in C. The course is required of majors, but most of the course’s students are non-majors. Each year, roughly one half of CS 50’s students continue on to a second course in computer science (CS 51). Although some students (10%) enter CS 50 with “a lot” of programming experience (e.g., AP Computer Science), most students describe themselves as having “a little” (46%) or “none” (44%), per Figure 1. A course with nearly 300 undergraduates and over 20 teaching fellows (TFs), most of whom are undergraduates themselves, CS 50 demands an average of 15 hours of work per week of students and TFs alike, much of which is spent on problem sets. For guidance on these problem sets, students turn not only to lectures and “sections,” classes led by TFs, but also to “office hours,” blocks of time during which

students can receive one-on-one help in a computer lab with their code on a screen and a TF by their side. Fifteen years ago, students signed up for such help by writing their name on a sheet of paper taped to the lab’s wall. Eventually, that sheet of paper evolved into a whiteboard, but the process today remains largely the same. It works, and it is simple. But it is not terribly efficient, as such intimate help does not scale very well. This past fall, up to 15% of our nearly 300 students would sometimes drop by office hours at once. Even with as many as 4 TFs on duty, our throughput under such loads tends not to be high, as demand for our help quite often exceeds our capacity. The result for students is not only long wait times but also frustration, if not an inclination simply to give up on some bug. Nor are these office hours always convenient, particularly for students who have short but show-stopping questions for whom the walk across campus takes longer than actually asking those questions. And we daresay this lab in a basement is not a particularly fun place for students (or teachers) to spend evenings. We do, of course, field questions by email, but our response time is not always immediate, given that TFs have studies of their own. And email itself introduces latency, with either the student or TF waiting for the other to reply to some thread. And so this past fall we decided to “virtualize” some office hours, whereby students could meet online with TFs instead of in person in order to discuss material or troubleshoot bugs. Our overarching goals were efficiency and convenience. We hypothesized that, with the right software and multiple windows, TFs could field questions from more students at once. And we further conjectured that, by making it easier to meet up, we could lower two bars: to students asking for help and to TFs offering the same, particularly at odd hours. We hoped, ultimately, to encourage interaction among TFs and students, albeit virtual. Based on surveys of students and discussions with TFs at term’s end, we ultimately judged our experiment with virtual office hours a success, but only for subsets of both groups. Although most students liked the idea of virtual office hours, only 55% actually logged on at least once. However, only 62% of students dropped by physical office hours at least once, so 55% is not necessarily low. Among TFs, reactions were mixed. Some outright preferred in-person interactions with students, whereas others valued the ability to work with students from home, even into the night, especially while multitasking. In the end, the audience for virtual office hours proved entirely self-selecting. Those students who liked the online experience tuned in, whereas those who did not like or did not need this particular resource tuned out. In the section that follows, we describe our approach to virtual office hours as well as the software we chose for our virtual lab. In Sections 3 and 4, we cite related work and alternatives, including approaches we and others have used in the past. In Section 5, we elaborate on our deployment of virtual office hours, thereafter presenting in Section 6 the results of our experiment. We conclude in Section 7 and propose future experiments. A link to CS 50’s website appears in this paper’s appendix.

2.

IMPLEMENTATION DETAILS

One-on-one face time between student and teacher is undoubtedly valuable, and so we did not opt to replace physical hours outright with virtual. We instead supplemented the physical with virtual. Instead of offering 2 physical hours per week, each TF instead offered 2 physical hours and 1 virtual. For our virtual lab, we first considered the obvious: email, online forums, and instant messengers. Via trial site licenses, though, we then turned to Elluminate Live! [5] and WebEx [12], both of which offer not only the ability to chat via instant messaging and VOIP, but also the ability to see and share control of participants’ screens (e.g., code in students’ terminal windows). We ultimately adopted Elluminate for two reasons. We found its learning curve to be lower, requiring minimal training of students and staff. Short demonstrations plus short handouts with screenshots sufficed. And we found its interface a better analog for our existing workflow. Not only does Elluminate allow students to “raise their hands” by clicking an icon, it also keeps track hands’ order, just like the actual lab’s whiteboard. With nearly 300 students, we appreciated the tool’s ability to maintain order and fairness. Traditional instant-messaging clients, by contrast, would be more likely to overwhelm us with windows anytime multiple students started chatting at once. Elluminate is essentially a glorified chatroom that allows participants to chat not only via instant messaging but also via VOIP. More importantly, though, the software allows students and TFs to see or even share control of each other’s screens (subject to both parties’ consent). As a result, not only can a TF draw a student’s attention to some line of code by controlling the student’s mouse from afar, the TF can even start typing in some window on the student’s computer (e.g., to jot down some pseudocode for subsequent reference). Similarly can TFs share their own screens (even without relinquishing control) in order to demonstrate something themselves. Figure 2 depicts Elluminate’s overall interface. Upon logging into Elluminate, students can “raise their hands” by clicking an icon. Numbers indicating their positions in line then appears next to their names in the list of participants. TFs appear in that same list in boldface as “moderators.” Only TFs can take control (if granted permission) of students’ screens and pull students into “breakout rooms” where correspondence and screen-sharing is private. For privacy’s sake, students need not share their entire desktop either; the software allows them to share individual windows (e.g., terminal windows alone). Written in Java, Elluminate itself is cross-platform. Whether students themselves run Windows, Mac OS, or Linux is no matter; all that’s required is a free Java Virtual Machine (JVM). To log into Elluminate, students only need visit a particular webpage; their browser should download any requisite JARs. The software connects participants to a centralized server, thereby avoiding problems that tend to arise because of firewalls and NAT when clients instead try to connect directly to each other. It’s worth noting that Elluminate includes features beyond what we use, including the ability to show PowerPoint slides, conduct polls, and draw on its “whiteboard.” Links to a recorded demonstration of Elluminate and a free (three-person) version of Elluminate appear in this paper’s appendix.

Figure 2: Elluminate’s interface consists primarily of: a list of participants that indicates the order in which students have raised their hands virtually (top-left); a chat window where students and TFs alike can communicate (bottom-left); and a whiteboard where students and TFs can see and/or share control of each other’s screens (right). In the above, Jane’s hand is raised, followed by John’s, per the numbers next to those names.

3.

RELATED WORK

4.

ALTERNATIVES

Elluminate is not quite an LMS (learning management system) but, rather, CSCW (computer supported cooperative work) software inasmuch as it allows teachers and students to tackle together, in our case, problems that the latter are having with code. As Schullo has noted, such software facilitates “synchronous learning” [11]. Unlike email and online forums, which lend themselves to asynchronous exchanges, Elluminate caters to real-time instruction. To be sure, virtual office hours themselves are not new, though relatively little literature covers precisely the subject. Kohorst and Cox have documented their experience with virtual office hours in a biochemistry course [7] as have Myers et al. for statistics and economics [8]. Related in spirit to Elluminate, meanwhile, is TechTalk by Johnson et al., a Webbased tool designed to facilitate scientific and mathematical collaboration online between students and teachers [6]. Of course, interactions online between teachers and students are certainly common in large courses, where listservs and forums often supplement or supplant more intimate meetings, as Preston and Shackelford have noted [9]. Distance-ed courses, meanwhile, are defined by their execution online.

Far more numerous than studies on virtual office hours are implementations of tools that facilitate the same. Perhaps the most obvious ones are email, online forums, and instant messaging clients. For years has CS 50 utilized email and forums to field questions from students, and the course has even experimented with an instant messaging “bot” for automated answers to frequently asked questions. Some instant messaging clients (e.g., Windows Messenger) actually include the ability to request control of another’s screen, but quite often have we found that firewalls and other hurdles get in the way of those tools. And they are not often crossplatform. WebEx, on the other hand, is a worthy contender, inasmuch as it offers most of the features that Elluminate does and even allows participants to tune in (to actual conversations) via phone. Unlike Elluminate, though, it does not appear to keep track of the order in which participants’ hands go up in the air. For us (and fairness’s sake), that was reason alone to turn to Elluminate. But just as influential were our first impressions of Elluminate and WebEx when compared side by side. Quite simply, we found Elluminate’s interface more self-explanatory, at least for our purposes. That, too, was enough to convince us to try it.

To be fair, WebEx markets itself quite clearly to businesses. It is not nearly as focused on target audiences like ours. Elluminate, by contrast, comes in an academic edition, and the software itself is tailored to online instruction. More similar in spirit to Elluminate, then, is software from Blackboard [3] that similarly enables students and teachers to congregate online. For reasons of cost, though, we shied away from that option. Wimba Pronto [13] also supports instant messaging and application sharing but only on Mac OS and Windows. Given that some students and some labs on campus run Linux, we preferred Elluminate’s reliance on JVMs. WiZiQ, meanwhile, offers Flash-based “Virtual Classrooms” for free [2], but screen-sharing requires that participants manually install third-party drivers. Dimdim offers its own “Virtual Classroom” [4], and Adobe offers Acrobat Connect Pro [1], but both of these tools allow only presenters (i.e., TFs but not students) to share applications and desktops. Yet another option for virtual classrooms is Saba Centra [10].

50% Virtual 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% "never" "rarely" "often" "constantly" Physical

5.

DEPLOYMENT

Inasmuch as Elluminate is effectively Web-based (at least its initial installation), deployment simply required that we add to the course’s website a link to a “session” (i.e., instance) that we configured at term’s start via Elluminate’s administrative console. Although Elluminate allows you to “invite” participants to a session with user-specific links, we found it simpler for students and TFs alike to integrate one link into the course’s website.∗ Our training of TFs and students, meanwhile, was minimal but, even in retrospect, sufficient. Although the TFs did spend an hour at term’s start practicing their roles as moderators, students were only offered a quick demonstration of the tool during lecture along with a short handout with screenshots. After that, most students found the software quite self-explanatory, with only 5% of students reporting any technical difficulties.

Figure 3: Based on surveys of students, we found that virtual office hours were attended nearly as frequently (or infrequently) as physical, with 14% of students attending the virtual and 21% the physical “often.” But not all reactions were positive. One student noted: “They are not as effective as the physical ones. The [TFs] can’t really give their full attention to the student.” Another explained: “They are helpful for small questions, but not for understanding larger concepts.” In the words of a third, “I really like the idea. I’m just more comfortable with someone physically there.” Explained others, “nothing beats physical office hours,” but virtual were nonetheless a “good substitute.” On the whole, we found that virtual office hours were attended nearly as frequently (or infrequently, if one prefers the glass half-empty) as physical. Per Figure 3, 21% of students claimed to have attended physical office hours “often,” while 14% of students claimed to have attended virtual the same. To be sure, these results are more qualitative than quantitative. But inasmuch as our goal was to involve students themselves in the evaluation of this experiment, it was precisely their anecdotes that we wanted to hear. Worthy of note are some unexpected lessons we learned. Although, on the whole, Elluminate’s interface did allow us to re-create virtually our in-person workflow, it is by no means perfect. Screen-sharing, even on campus or broadband, suffered from latency that sometimes decelerated oneon-one help. The software also did not natively allow TFs to be in multiple breakout rooms at once, the result of which is that juggling multiple students was not as easy as hoped. Our workaround was for TFs to sign in as moderators multiple times, with one window per student. Even so, wait times online sometimes matched or exceeded those in the lab. Similarly did the software not make it easy to alert students to a TF’s presence, once it was their turn for help. Quite often would students raise their hands virtually and then background or minimize Elluminate, the result of which was missing their turn. “When a [TF] greets you,” lamented one student, “it’s easy to miss.” Unfortunately, the company has not responded to repeated requests for Elluminate to enable moderators to beep or flash participants’ screens.

6.

RESULTS

With 23 TFs, we were generally able to offer 23 virtual office hours per week and 46 physical, spread out to accommodate TFs’ and problem sets’ schedules. But we adjusted the schedule over time based on demand and attendance. At term’s end, we surveyed all students on their experience, if any, with virtual office hours, asking them not only how often they attended but also what they thought of the experience. Most students’ reactions to our virtualization of office hours were positive. In the words of one student, “I think they’re a great idea. Help is just as good, and I find I think much better when I’m not in the stress of the physical office hours.” In the words of another, “Virtual office hours are definitely more convenient. And while I am waiting to be helped, I can just keep coding.” The overall consensus seemed to be that “they’re great for shorter questions.” Of course, as is the case with many technologies, enthusiasm does not necessarily mean adoption. Not uncommon were admissions like “I think it’s a really cool idea, I just haven’t gotten around to trying it out” and “A neat idea. Haven’t used it though.”
∗ As a result, though, we were not able to log usage as easily, and so we measured usage via surveys of students.

80% 70% 60% 50% 40% 30% 20% 10% 0% prefer physical long waits technical difficulties

Meanwhile, to address the problem of wait times, we will likely consolidate virtual office hours into fewer blocks, each staffed with more TFs at once. We may also dedicate a separate Elluminate session to the shortest of questions so that we can continue to focus on the students who need our support most, even while fielding more students’ questions.

APPENDIX
The URL of CS 50’s website is http://www.cs50.net/. A recorded demonstration of Elluminate’s interface can be found at http://www.elluminate.com/demo/. A free (three-person) version of Elluminate is available at: http://www.elluminate.com/vroom/.

Figure 4: Among the subset of students (166 out of 282) who did not tune in for virtual offices very much or at all, 74% explained that they simply preferred physical, 42% explained that, at least once, they had to wait too long, and 5% admitted they couldn’t figure out how to tune in at all. Some students cited more than one of these reasons. Our current workaround is to set a 1-second timer, a side effect of which is to beep students’ screens. Ultimately, our goals of efficiency and convenience were only realized in part. Virtualization of office hours indeed enabled us to help multiple students at once (albeit via a workaround), but the process was not as rapid as we would have liked. Students are not always the fastest of typists and did not always realize there was a TF waiting for their next line of text. And, as students themselves found, virtual office hours proved more useful for diagnosis of small bugs than for overhauls of code. Among the subset of students (166 out of 282) who did not tune in for virtual offices very much or at all, 74% explained that they simply preferred physical, 42% explained that, at least once, they had to wait too long, and 5% admitted they couldn’t figure out how to tune in at all, per Figure 4. For many students and staff, though, the ability to hold virtual office hours, even while multitasking, was absolutely convenient. We found ourselves logging in at all hours because it was so easy to do.

ACKNOWLEDGEMENTS
Many thanks to Harvard University’s iCommons team for their support of this experiment and to Henry H. Leitner for introducing us to Elluminate. And many thanks to Fall 2007’s 282 students and 23 teaching fellows, without whom this experiment would not have been possible.

REFERENCES
[1] Adobe Systems Inc. Acrobat Connect Pro. http: //www.adobe.com/products/acrobatconnectpro/. [2] authorGEN Technologies. WiZiQ Virtual Classroom. http://www.wiziq.com/Virtual_Classroom.aspx. [3] Blackboard Inc. Blackboard Learning System. http://blackboard.com/products/academic_suite/ learning_system/. [4] Dimdim Inc. Virtual Classroom. http://www.dimdim. com/products/dimdim-virtual-classroom.html. [5] Elluminate, Inc. Elluminate Live! http://www.elluminate.com/. [6] J. Johnson, Y. N. Lakshman, T. T. Hewett, T. Souder, T. Fitzgerald, S. Donegan, and P. Morgovsky. Virtual Office Hours using TechTalk, a Web-Based Mathematical Collaboration Tool. SIGCSE Bull., 30(3):130–133, 1998. [7] K. Kohorst and J. R. Cox. Biochemistry and Molecular Biology Education, volume 35. May 2007. [8] S. C. Myers, D. Bishop, S. Rajamany, and J. Kelley. Virtual Office Hours: Tutoring Distance Students in Mathematics, Statistics and Economics. In Ohio Commons for Digital Education (OCDE) Conference, 2004. [9] J. A. Preston and R. Shackelford. A System for Improving Distance and Large-Scale Classes. SIGCSE Bull., 30(3):193–198, 1998. [10] Saba Software, Inc. Saba Centra. http://www.saba.com/solutions/academic/. [11] S. Schullo. A Tale of Two Systems Elluminate Live vs. Macromedia Breeze. In 22nd Annual Conference on Distance Teaching and Learning, Madison, Wisconsin, 2006. [12] WebEx Communications, Inc. WebEx. http://www.webex.com/. [13] Wimba, Inc. Wimba Pronto. http://www.wimba.com/products/wimba_pronto/.

7.

CONCLUSION

On the whole, we judged our integration of virtual office hours into CS 50 a net positive. We make no claims that virtual office hours constitute some revolutionary new technology but, rather, that they represent another, imperfect but still useful, resource that we now make available to students. They certainly facilitate an alternative form of interaction with students. But we recognize that this form might not work nearly as well in other courses, particularly those that are not already heavily keyboard-based. We have since deployed virtual office hours in other courses as well, particularly at Harvard Extension School and Harvard Summer School, both of which offer courses that are themselves entirely online. We have also used Elluminate to offer virtual sections for distance-ed courses, with TFs leading online presentations using VOIP for their voiceover. We even record those online sessions for students’ subsequent review.