Steven J. Castellucci

Product Redesign for a Cross-Disciplinary Userbase

Contents:

1. Overview
2. Process
   1. Who Are Our Users?
   2. What to Teach?
   3. How to Teach?
   4. Conflicts and Compromises
3. Outcomes, Insights, and Deliverables

Overview

Summary: I served as the Course Director (i.e., course designer, administrator, manager, and instructor) of a computer fundamentals course within the Electrical Engineering and Computer Science (EECS) department. The course had a typical enrolment of about 800 students per term, divided into 6 course sections, taught by 4‑5 instructors, and supported by 6‑10 teaching assistants (TAs). One of my responsibilities was to redesign the course by updating its content to engage its diverse, cross-disciplinary audience.

Users and Stakeholders: The users (i.e., students) hailed from multiple departments (mainly Biology, but others had yet to be determined), with diverse learning backgrounds and goals for the course. The stakeholders included my fellow course instructors, my Director, and representatives from the students' respective departments.

Pain Points: This course had outdated content. Students lost interest, and the Biology department considered removing this EECS course as a requirement in their program. That would have significantly decreased our enrolment, resulting in significant lower revenue for the EECS department. Additionally, course assignments had not changed in several years. Students would blatantly plagiarize work and go unpunished because of the vast number of incidents.

Scope and Constraints: The first lecture was in four weeks. Although course materials could be developed throughout the term, course details (e.g., lecture topics, presentation format, number and type of assignments) needed to be finalized for the syllabus before the course began.

Methods Used: Guerrilla user interviews, stakeholder interviews, guerrilla collaborative design, content audit, competitive analysis, qualitative surveys, quantitative surveys, metrics analysis, and log analysis.

Process

Who Are Our Users?

What to Teach?

To determine which topics should be included in the redesigned course, I first performed a content audit of the current topics (summarized in the table below). I also used informal user interviews with students, and engaged stakeholders in an informal collaborative design endeavour. I knew that students would be hesitant to give negative feedback if I were to ask, “Is there anything wrong with this course?”, as I was the one tabulating their grades. Instead, I would ask, “What would you like to see in this course?” to illicit feedback. I learned that students wanted to better understand the computer terminology used when buying new hardware, as well as updated real-world examples of the concepts taught. The lab manual featured the “new” Windows XP, which had been released 15 years prior.

The previous Course Director, John, and I had informally discussed improvements to this course. He wanted to add modules on Wi-Fi and image editing. Additionally, my Director wanted the modules on history of computing and digital data representation removed in favour of spending more time on other topics. I agreed that time devoted to digital data representation should be reduced, but not eliminated. I justified its inclusion with an example of how a simple looking formula in Excel yields an incorrect result: =1*(0.8-0.5-0.3) returns “5.55112E-17”, instead of zero. The binary representation of these numbers in Excel causes a loss of precision in the calculation (i.e., a round-off error). I argued that students should have a basic understanding of digital data representation to know that such errors can occur, and my Director agreed.

With a tentative list of revised course topics, I reached out to my representatives in client departments. Kinesiology agreed with my suggested changes, but I did not receive a response from Psychology. Chemistry wanted to remove the modules on Boolean logic and digital data representation, and add modules on presentation skills, general understanding of the world-wide web, advanced word processing with Word (e.g., paragraph styles, captions, cross-references, updating fields), and big data mining using databases. Biology agreed with the suggested additions, but wanted to keep Boolean logic. Biology also wanted topics on R programming, computer system administration (i.e., file systems, maintenance, security, backup strategies), advanced pixel and vector graphics, and web authoring (i.e., HTML, scripting, and CSS). It was impractical to cover all topics in a single introductory course, so I selected a subset that I felt would form a cohesive curriculum.

How to Teach?

Redesigning this course gave me an opportunity to update pedagogical methods, as well as the topics taught. In informal user interviews, students often criticized the traditional format of lecturers reading from slides, bulky expensive textbooks, assigned readings, and time-consuming weekly assignments with little weight on the final grade. I had taught the course the previous year and identified additional pain points from the reasons that students gave for skipping lectures, or missing assignment deadlines (e.g., “I had a lab scheduled at the same time”, “I don't have time to study because of my long, crowded commute”, “I had questions, but I didn't know how to get help”). From that, I identified the desirable product features:

1. Engaging, relevant lectures
2. Absence of physical textbooks
3. Audio or video supplemental materials that replace the lab manual and assigned readings
4. Fewer assignments, each with significant weight towards the final grade
5. Asymmetric learning to accommodate students' busy schedules
6. Getting help with an assignment is easier than finding a solution to copy

To identify possible solutions, I used a method similar to a competitive analysis. Although I did not see other courses or faculty members as “competition”, I did want to explore what pedagogical methods my peers used. A new faculty member, James, introduced me to the “flipped classroom” methodology: Instead of students learning in lecture and completing assignments on their own time, they consume course materials on their own time (feature (5) above) and complete activities in lectures. We tried this in a separate computer engineering course. The flipped classroom sessions were semi-weekly, and the activities had little weight on the final grade. Towards the end of the term, students would skip the flipped classroom session to work on projects in other courses. For the computer fundamentals course, I decided to use the flipped classroom strategy, but with less frequent sessions, and larger activities to encourage student participation.

In other courses that I taught, I had already started recording my lectures and posting them for students to watch or review after lectures, and I continued to do so with this course to facilitate feature (5) above. I also structured each topic around something relatable to students, and promoted discussions in lecture (feature (1) above). For the module on computer hardware, I role played buying a new computer. On the topic of digital data representation, I discussed how online videos are encoded to load faster. And for computer networking, I discussed how to look for an internet service provider, and how to protect against wireless identity theft. For each module, I used multiple metaphors and comparisons in my explanations. I felt this increased the opportunity for students to relate to the topic.

The best things about this course are that the professor is extremely clear in explaining the information to the students, and that he provides a lecture recording. With out [sic] those recordings I wouldn't have been able to excel in the class the way I did. I am someone who is constantly ill and learns best when I hear the information being explained with various comparisons.
 — Anonymous survey response

Throughout the term, I produced a series of online videos presenting course topics, such as Excel formulas and charting, operating systems, file systems and backing up data, presentation skills, and image editing. Posting the videos to YouTube made them easily accessible via a laptop, tablet, or smartphone during one's commute. I also developed assignments that allowed students to apply the knowledge presented in the videos. I structured the assignments to be easily modifiable each term, and to require students to integrate something personal (e.g., student number, student ID, or a selfie) in order to make each submission unique. This facilitated easy identification of cheating, and discouraged sharing of assignment submissions online.

 Course Videos on YouTube

Using a flipped classroom approach with online videos and corresponding assignments implemented features (2) and (3) above. And I scheduled the “flipped classroom” lectures around the deadlines for the four new course assignments that replaced the previous weekly assignments (feature (4) above). This allowed students scheduled time to work on the assignment, or ask questions about the assignment (feature (6) above).

The flipped classroom videos that my prof made were really good. The revamp of this course was a massive improvement over when I took it two years ago.
 — Anonymous survey response

Conflicts and Compromises

The majority of stakeholders agreed that the topic of computing history was unnecessary and should be removed to make room for other topics. However, two senior instructors insisted on keeping the topic. One of them compared students learning about the history of computing to them “eating their vegetables”, saying, “they don't know what's good for them” — a sentiment that I did not share. I agreed that the other course topics deserved as much lecture time as feasible, but I also did not want a rift with the two of the most senior faculty in the department. I decided to compromise. I could spare at most a single lecture (approximately 50 minutes) without negatively impacting the other topics. Thankfully, all the stakeholders reluctantly agreed to this.

But how to make the history of computing more palatable to our users, the students? Anecdotally, the majority of students in the course had been young women. Additionally, the university was generally known to be LGBTQ-friendly. I decided to modify the history of computing module to highlight the contributions of women (e.g., Ada Lovelace, Margaret Hamilton, and the original ENIAC programmers) and LGBTQ individuals (e.g., Alan Turing, Lynn Conway). Although students still rated the computing history as the least useful topic, the feedback with regards to the change were overwhelmingly positive. One student even reached out with praise via email.

 Women in Computing History Lecture

Outcomes, Insights, and Deliverables

Towards the end of the course, I used a survey with quantitative and qualitative components to gather student feedback. I administered it via the learning management system (LMS), which automatically anonymized the responses and summarized the results. Approximately 24% (n=149) of students enrolled in the course opted to participate in the survey. Charts and quotes from the student responses appear throughout this document. The vast majority of students found the online videos helpful, and found the Excel and Word modules the most useful.

I completed this project towards the end of my three-year teachng contract, so I did not have the opportunity to implement any subsequent changes to the course. If I had had the opportunity, I would have produced more videos, expanded the pool of course assignments, and added course topics on statistical analysis (t-tests/ANOVA) in Excel and bibliographic tools in Word. The suggested topics of R programming, big data analysis, and graphical web design would be best served in more advanced or dedicated courses.

In summary, here are the outcomes and insights from the product redesign:

• Updated knowledge of the course's clients; the EECS department believed Chemistry and Psychology comprised a larger audience of the course than Kinesiology, but my analysis showed that the opposite was true
• Over 81% of students approved of the number of assignments, and 94% of students responded favourably to the new course videos
• Student participation in the course increased 47%, as measured by analysing metrics for assignment submissions
• Exam results increased 12%, suggesting better retention of the subject matter
• Decreased number of academic honesty violations (e.g., cheating, plagiarism), low enough to pursue through the official process; however, this created more administrative work, as each case could take months to fully investigate (a new pain point)
• Buy-in from the Biology department for the continued requirement of this course in their program, and thus, no decrease in funding for the EECS department

Here are the project deliverables I provided to the EECS department, my fellow instructors, and my TAs:

• Revised syllabus, detailing the new curriculum topics, learning objectives, number and weight of assignments, and policy changes
• Complete migration to an LMS to manage course content and logistics
• Online videos to implement the flipped classroom methodology and to facilitate asymmetric learning [Playlist on YouTube]
• Accurately transcribed video subtitles to facilitate accessibility for students who are deaf or hard of hearing
• Updated slide decks with verified links and references
• New assignments to thwart student plagiarism
• Assignment guide for future Course Directors, detailing assignment modifications to discourage plagiarism but maintain learning objectives and difficulty level
• Instructional videos for TAs to learn new assignment grading rubrics
• Instructional videos for fellow instructors to learn how to use the LMS to administer assignments, manage workflows, and enter grades
• Inclusive approach to presenting the history of computing [Lecture Recording]

The changes made to this course from last year were very beneficial. After discussing the current course with friends who took this course last year, I feel huge improvements were made.
 — Anonymous survey response