Product Redesign for a Cross-Disciplinary Userbase

a large lecture hall
  1. Overview
  2. Process
    1. Who Are Our Users?
    2. What to Teach?
    3. How to Teach?
    4. Conflicts and Compromises
  3. Outcomes, Insights, and Deliverables


Summary: I served as the Course Director (i.e., course designer, administrator, manager, and instructor) of a computer fundamentals course within the Electrical Engineering and Computer Science (EECS) department. The course had a typical enrolment of about 800 students per term, divided into 6 course sections, taught by 4‑5 instructors, and supported by 6‑10 teaching assistants (TAs). One of my responsibilities was to redesign the course by updating its content to engage its diverse, cross-disciplinary audience.

Users and Stakeholders: The users (i.e., students) hailed from multiple departments (mainly Biology, but others had yet to be determined), with diverse learning backgrounds and goals for the course. The stakeholders included my fellow course instructors, my Director, and representatives from the students' respective departments.

Pain Points: This course had outdated content. Students lost interest, and the Biology department considered removing this EECS course as a requirement in their program. That would have significantly decreased our enrolment, resulting in significant lower revenue for the EECS department. Additionally, course assignments had not changed in several years. Students would blatantly plagiarize work and go unpunished because of the vast number of incidents.

Scope and Constraints: The first lecture was in four weeks. Although course materials could be developed throughout the term, course details (e.g., lecture topics, presentation format, number and type of assignments) needed to be finalized for the syllabus before the course began.

Methods Used: Guerrilla user interviews, stakeholder interviews, guerrilla collaborative design, content audit, competitive analysis, qualitative surveys, quantitative surveys, metrics analysis, and log analysis.


Who Are Our Users?

I wanted to identify the specific departments that were recommending or requiring this computer fundamentals course to its students. I knew that Biology was a large “client”, but I did not know to what extent, or how many other large clients there were. EECS kept log files to record student enrolment in each of its courses. My Director mentioned that some departments had changed degree requirements about three years prior, so I only used logs from the past three years. I wrote a Java program to parse each students' home department, based on their declared major, and to count the total occurrences of each one.

My findings confirmed that Biology majors comprised a large part of the course's audience, followed by Kinesiology, Psychology, Chemistry, and various business-related majors (e.g., Business Administration, Economics). With this information, I reached out to the respective departments' Directors, introduced myself and this project, and asked for a point-of-contact representative who could provide me with input and feedback on my changes.

Biology 33%, Kinesiology 27%, Psychology 10%, Chemistry 8%, Business 7%, others 15%

What to Teach?

To determine which topics should be included in the redesigned course, I first performed a content audit of the current topics (summarized in the table below). I also used informal user interviews with students, and engaged stakeholders in an informal collaborative design endeavour. I knew that students would be hesitant to give negative feedback if I were to ask, “Is there anything wrong with this course?”, as I was the one tabulating their grades. Instead, I would ask, “What would you like to see in this course?” to illicit feedback. I learned that students wanted to better understand the computer terminology used when buying new hardware, as well as updated real-world examples of the concepts taught. The lab manual featured the “new” Windows XP, which had been released 15 years prior.

The previous Course Director, John, and I had informally discussed improvements to this course. He wanted to add modules on Wi-Fi and image editing. Additionally, my Director wanted the modules on history of computing and digital data representation removed in favour of spending more time on other topics. I agreed that time devoted to digital data representation should be reduced, but not eliminated. I justified its inclusion with an example of how a simple looking formula in Excel yields an incorrect result: =1*(0.8-0.5-0.3) returns “5.55112E-17”, instead of zero. The binary representation of these numbers in Excel causes a loss of precision in the calculation (i.e., a round-off error). I argued that students should have a basic understanding of digital data representation to know that such errors can occur, and my Director agreed.

With a tentative list of revised course topics, I reached out to my representatives in client departments. Kinesiology agreed with my suggested changes, but I did not receive a response from Psychology. Chemistry wanted to remove the modules on Boolean logic and digital data representation, and add modules on presentation skills, general understanding of the world-wide web, advanced word processing with Word (e.g., paragraph styles, captions, cross-references, updating fields), and big data mining using databases. Biology agreed with the suggested additions, but wanted to keep Boolean logic. Biology also wanted topics on R programming, computer system administration (i.e., file systems, maintenance, security, backup strategies), advanced pixel and vector graphics, and web authoring (i.e., HTML, scripting, and CSS). It was impractical to cover all topics in a single introductory course, so I selected a subset that I felt would form a cohesive curriculum.

Existing Curriculum Redesigned Curriculum
History of Computing (see Conflicts and Compromises below)
Boolean Logic Boolean Logic (abridged)
Digital Data Representation (numbers, text) Digital Data Representation (updated to include audio, images, and video)
Computer Components Computer Components (updated technology, and added “Buying a Computer” roleplay activity)
Operating System Concepts (CPU scheduling, process and memory management) Operating System Concepts (added virtualization, file systems, file operations, file permissions, backup strategies, and walkthroughs of Windows, macOS, and Linux file explorers and command-line terminals)
Databases (removed due to technical limitations)
Computer Networking Computer Networking (added Wi-Fi, NFC, and protecting against security vulnerabilities)
The Internet The Internet (added cloud computing and storage)
The WWW The WWW (added web security risks and countermeasures, CSS, and JavaScript coding activity with user-defined functions, if-statements, and event handling)
  Image Editing (selections, transformations, levels, layers, filters, colour replacement, exporting)
  Advanced Word Processing (styles, typography, figures, captions, cross-references, fields, tracking changes)
  Presentation Skills (outline view, design themes, master layouts, presenter view, point-form slides, use animations sparingly, audience atmosphere, presentation anxiety)

How to Teach?

Redesigning this course gave me an opportunity to update pedagogical methods, as well as the topics taught. In informal user interviews, students often criticized the traditional format of lecturers reading from slides, bulky expensive textbooks, assigned readings, and time-consuming weekly assignments with little weight on the final grade. I had taught the course the previous year and identified additional pain points from the reasons that students gave for skipping lectures, or missing assignment deadlines (e.g., “I had a lab scheduled at the same time”, “I don't have time to study because of my long, crowded commute”, “I had questions, but I didn't know how to get help”). From that, I identified the desirable product features:

  1. Engaging, relevant lectures
  2. Absence of physical textbooks
  3. Audio or video supplemental materials that replace the lab manual and assigned readings
  4. Fewer assignments, each with significant weight towards the final grade
  5. Asymmetric learning to accommodate students' busy schedules
  6. Getting help with an assignment is easier than finding a solution to copy

To identify possible solutions, I used a method similar to a competitive analysis. Although I did not see other courses or faculty members as “competition”, I did want to explore what pedagogical methods my peers used. A new faculty member, James, introduced me to the “flipped classroom” methodology: Instead of students learning in lecture and completing assignments on their own time, they consume course materials on their own time (feature (5) above) and complete activities in lectures. We tried this in a separate computer engineering course. The flipped classroom sessions were semi-weekly, and the activities had little weight on the final grade. Towards the end of the term, students would skip the flipped classroom session to work on projects in other courses. For the computer fundamentals course, I decided to use the flipped classroom strategy, but with less frequent sessions, and larger activities to encourage student participation.

In other courses that I taught, I had already started recording my lectures and posting them for students to watch or review after lectures, and I continued to do so with this course to facilitate feature (5) above. I also structured each topic around something relatable to students, and promoted discussions in lecture (feature (1) above). For the module on computer hardware, I role played buying a new computer. On the topic of digital data representation, I discussed how online videos are encoded to load faster. And for computer networking, I discussed how to look for an internet service provider, and how to protect against wireless identity theft. For each module, I used multiple metaphors and comparisons in my explanations. I felt this increased the opportunity for students to relate to the topic.

The best things about this course are that the professor is extremely clear in explaining the information to the students, and that he provides a lecture recording. With out [sic] those recordings I wouldn't have been able to excel in the class the way I did. I am someone who is constantly ill and learns best when I hear the information being explained with various comparisons.
 — Anonymous survey response

Throughout the term, I produced a series of online videos presenting course topics, such as Excel formulas and charting, operating systems, file systems and backing up data, presentation skills, and image editing. Posting the videos to YouTube made them easily accessible via a laptop, tablet, or smartphone during one's commute. I also developed assignments that allowed students to apply the knowledge presented in the videos. I structured the assignments to be easily modifiable each term, and to require students to integrate something personal (e.g., student number, student ID, or a selfie) in order to make each submission unique. This facilitated easy identification of cheating, and discouraged sharing of assignment submissions online.

 Course Videos on YouTube

Using a flipped classroom approach with online videos and corresponding assignments implemented features (2) and (3) above. And I scheduled the “flipped classroom” lectures around the deadlines for the four new course assignments that replaced the previous weekly assignments (feature (4) above). This allowed students scheduled time to work on the assignment, or ask questions about the assignment (feature (6) above).

The flipped classroom videos that my prof made were really good. The revamp of this course was a massive improvement over when I took it two years ago.
 — Anonymous survey response

Conflicts and Compromises

The majority of stakeholders agreed that the topic of computing history was unnecessary and should be removed to make room for other topics. However, two senior instructors insisted on keeping the topic. One of them compared students learning about the history of computing to them “eating their vegetables”, saying, “they don't know what's good for them” — a sentiment that I did not share. I agreed that the other course topics deserved as much lecture time as feasible, but I also did not want a rift with the two of the most senior faculty in the department. I decided to compromise. I could spare at most a single lecture (approximately 50 minutes) without negatively impacting the other topics. Thankfully, all the stakeholders reluctantly agreed to this.

But how to make the history of computing more palatable to our users, the students? Anecdotally, the majority of students in the course had been young women. Additionally, the university was generally known to be LGBTQ-friendly. I decided to modify the history of computing module to highlight the contributions of women (e.g., Ada Lovelace, Margaret Hamilton, and the original ENIAC programmers) and LGBTQ individuals (e.g., Alan Turing, Lynn Conway). Although students still rated the computing history as the least useful topic, the feedback with regards to the change were overwhelmingly positive. One student even reached out with praise via email.

I just wanted to thank you for including all the women pioneers in our course. I don't know if its [sic] the course requirements or you choose to do so but I really appreciate it! As a women [sic], I've always wondered why not many women showed up in my textbooks or why I wasn't taught much about women as about men. I'm surprised at all the women involved with history of computing and I'm grateful that you chose to make women stand out in our course.
 — Nitnem G.
excellent 30%, very good 39%, neutral 19%, not interesting 10%, return to previous approach 1%

 Women in Computing History Lecture

Outcomes, Insights, and Deliverables

Towards the end of the course, I used a survey with quantitative and qualitative components to gather student feedback. I administered it via the learning management system (LMS), which automatically anonymized the responses and summarized the results. Approximately 24% (n=149) of students enrolled in the course opted to participate in the survey. Charts and quotes from the student responses appear throughout this document. The vast majority of students found the online videos helpful, and found the Excel and Word modules the most useful.

Most useful: Excel 64%, Word 36%, digital media 33%; Least useful: history of computing 63%, representing numbers 37%, representing text 37%
very helpful 64%, somewhat helpful 27%, neutral 3%, not helpful 2%, prefer lab manual 3%

I completed this project towards the end of my three-year teachng contract, so I did not have the opportunity to implement any subsequent changes to the course. If I had had the opportunity, I would have produced more videos, expanded the pool of course assignments, and added course topics on statistical analysis (t-tests/ANOVA) in Excel and bibliographic tools in Word. The suggested topics of R programming, big data analysis, and graphical web design would be best served in more advanced or dedicated courses.

In summary, here are the outcomes and insights from the product redesign:

Here are the project deliverables I provided to the EECS department, my fellow instructors, and my TAs:

The changes made to this course from last year were very beneficial. After discussing the current course with friends who took this course last year, I feel huge improvements were made.
 — Anonymous survey response