In May of 2016, our small but close knit team of 6 had completed our funding goal on Kickstarter to create our learn to code platform, Devslopes. Our funding milestones were structured and tiered by platforms. The first few milestones committed us to creating apps for the Apple ecosystem — specifically iOS, macOS, and tvOS. As the milestones progressed, we committed to creating an Android, Windows and Linux version respectively.
With the majority of our team having roots in iOS development, we started building our iOS and macOS app. We wanted to reach feature parody with widely known LMS systems as quickly as possible while validating our main assumption: Gamifying ed-tech would be the x-factor that boosts retention and student success. Coins could be earned through completing course content, and then used to redeem new courses and exclusive learning resources.
With a focus to create a gamified experience coupled with high quality learning content, we used out-of-the-box solutions when necessary. We implemented Firebase for our backend and relied on Apple’s built in systems for monetization (In-App-Purchases), analytics, and distribution.
10 months after launching, we learned that gamification actually had the opposite effect on student learning. On average, our students became more focused on aggregating coins through whatever means necessary just to redeem resources — not necessarily to utilize them. The majority of our users were indifferent on gamification, and some even found it a slight distraction to their content consumption. As our user base began to grow, Firebase also became restrictive as our learning paths became more complex. We also had difficulty using Firebase to deploy against custom marketing systems such as coupon codes and course redemptions, and were unfortunately at the mercy of Apple’s update process when we needed to ship critical updates.
We learned that on a fundamental level our students loved our content, teaching style, and the idea of our custom platform for content delivery. However, our main assumption on gamification proved false and our current systems were putting us into technical debt. Armed with new data, we set out to rebuild our platform for better scalability, cross platform functionality, and flexibility for monetization and distribution.
As the product manager and sole designer on the team, I crafted the app experience for both our existing users and on-boarding new users. I created branding guidelines for colors, typography, and design language. I also collected and utilized our current students feedback and usage data points across our existing native macOS app.
While our current macOS native version performed well, we needed to prioritize content organization and the overall learning experience. We also had to be mindful for 3 different user types created from previous pivots and their experience when migrating to our new platform.
Our goal was to declutter design flows that performed under par with our current version, and double down on a few key features that enhanced the learning experience. We needed to make it easy for our current users to understand our new platform while maintaining conversion rates for new users. Our project timeline from design to launch was four months.
Gathering Data & User Research
All of my research could be distilled into three main categories: Our tracking systems, our team, and our users.
We relied heavily on Apple’s systems to track our unit economics and conversion rates. Many new users discovered us through the macOS App Store, and the rate in which they became paying customers best reflected our true conversion rate for new users. We also had a decently large presence on Udemy, and utilized instructor analytics as a benchmark for session times.
We also placed an easy to use Feedback Bar in our macOS app that allowed students to submit feedback anywhere in the app. Session information was tracked and sent with every feedback message we received. These were sent directly to me, and I relied on them heavily when planning our 2.0 re-design. Our most frequent questions came from users not knowing where to download source code and where our new courses could be found.
Being a small team, all of our team members had daily interactions with users in varying capacities. I made a concerted effort to speak to each team member one-on-one and get their candid feedback on our platform. I asked where they felt our platform needed to improve the most and what they felt we did best. The overall tone was that our content navigation could be improved greatly while maintaining the ease of use. We also needed to improve communication with our students through the app. We had paying users that had survived a few pivots, and many were unsure if we had remembered their account status. That needed to be reflected quickly and easily for our users upon version 2.0.
In addition to aggregating feedback via the in-app Feedback Bar, I also sent out a survey to our students in exchange for a free course. Through the 800 responses, our biggest takeaway was that our Picture in Picture feature (PIP) could be improved upon, and that our students wanted certifications for completed courses and tracks.
I also spearheaded our 2.0 Beta Initiative, allowing a select 50-100 users to get early beta access to our 2.0 platform as we made progress. In exchange, they would provide feedback and follow beta testing steps when prompted. These power users were also vocal of the feedback reflected in the student-wide survey.
DEFINING A NORTH STAR METRIC
After analyzing all the feedback from the team and our amazing community of students and users, we decided that one metric should define our success: Session times spent learning in the app.
Our average session time across all students on Udemy was roughly 4 minutes. While we matched that in our native macOS app, our goal was to double it. As a product owner, I ensured all product features, enhancements, and the design structure of our 2.0 app would point to increasing session learning times.
Optimizing the Learning Experience
We started with the largest use case and reassessed how our students were jumping back in to their lessons. Users had to go through 2 screens to continue their courses upon launch, while having to process advertisements for new courses, and possibly being distracted with other courses they were also enrolled in.
macOS (version 1) Flow
To Continue Learning: Recently played menu item (Left), Recently played screen (Center), Video Player (Right)
The first iteration of our redesign heavily accounted for our various user types. The toggle at the top accounted for any previously purchased courses or legacy content. This iteration also featured highly graphical components and big icons.
I solicited early feedback from our team and a few users before proceeding. Users were excited about the left navigation, but the main content window was met with some initial confusion. A quick verbal explanation made the layout clear, but we needed to have users understand it at a glance. It was apparent that we were headed in the right direction, but needed to present the data in a more clear and concise manner. This is also when we decided to use Electron (instead of macOS native again). This removed a few more design constraints that were imposed previously.
Creating a clearer context
I noticed that Slack did an amazing job at subconsciously creating a hierarchy of importance with their users. With chat apps, users can easily become distracted, yet our experience with Slack has remained seamlessly focused. I wanted to deduce and instill those components in our product. Creating a hierarchy from left to right, with the main content window being the end of the pipeline worked perfectly for our use case.
Version 2 Final Main Layout
The final new layout checked every box on our list. Students could automatically pick up where they left off on their courses upon launch. No additional screens or clicks were needed to jump back into their courses, and users easily understood the context in which the information was presented.
Users had visibility to other courses in the left most panel, so switching context to a new course became secondary and less distracting. By rearranging our layout, student questions on where to find new courses and lessons decreased by 60%.
A Clear Path to Success
To keep students on track and continuously learning, we wanted their journey to be at the forefront at all times. Upon launch they were able to visually see their path to completion, rather than jumping into the video player screen to view the lesson list and be reminded of their progress.
Crafting the Lesson List
The lesson list should reflect our North Star Metric as fluidly as possible. Rather than merely listing tasks or lessons to complete, we added icons to each type (Quizzes, Tests, Videos, or Projects). This helped the students further visualize the various milestones on road to success while being able to collapse and expand content in context.
The lesson list also held the source code download link in our macOS version. Due to the lesson list in 2.0 being constantly visible, questions regarding where to download the lesson files virtually vanished post launch.
Screen Real Estate - Learning to Code
The majority of our students were learning on a laptop with no external monitor, causing them to vocalize their need for Picture in Picture (PIP). While we shipped an update with PIP for our macOS version, the process to use it was cumbersome in the context of managing the app, an IDE, and the PIP window.
Solution - Theater Mode
Instead of recreating the PIP experience for our 2.0 revamp, we created “Theater Mode,” which allowed the benefits of PIP while reducing the process to achieve the desired results. This resulted in a decrease of time it took for students to configure window sizes, and an increase in maneuverability between their IDE and our app.
Quizzes & Tests
We implemented quizzes and final tests as requirements to complete the Slope. Final tests were timed to reduce and discourage research while taking them. Framing tests in this way resulted in higher completion rates and more engaged learning. We also added certificates and reflected it as such in the lesson list. This gave users a stronger sense of purpose going through the lessons with a tangible goal in mind.
Over the few years of our existence, we had supported 3 different user types through various pivots. We had Kickstarter Backers who received lifetime perks, Users who had subscribed to a monthly plan, and students who purchased courses a la carte. Instead of parsing through these directly in the app, user account types were denoted and accounted for upon check out. The easiest way was to funnel all users (regardless of what type) through this process, and have it sorted via checkout page and prior to their app experience. This kept the experience consistent across all user types, and minimized development overhead.
DEVSLOPES 2.0 PROTOTYPE demo
Over a 4 month period we were able to design, develop, and ship our product addressing our biggest pain points. We decided to rebuild our app in Electron, opening up our product to Windows and Linux users. This allowed us to create a blazing fast, highly customizable video player with “Theater Mode” that addressed users pain points of watching videos while learning. This also relinquished design limitations and allowed us to optimize on creating a clearer learning track, decreasing confusion for lesson locations and source code files, while increasing average session times. We were also able to build our own API for full customization and flexibility.
The Devslopes 2.0 app was very well received by our community, and in its first month of launch it generated revenue for 5 months of runway.