Abstract

This report explores how course management and teaching can be optimized to enhance the success of graduate students at Florida Tech. An anonymous student questionnaire was used to elicit common trends and perceptions within the context of student engagement, learning outcomes and effective teaching practices. Graduate students rated their interactions with faculty and teaching practices as the main contributors to their engagement. Responses demonstrated a proclivity for learning and communicating with technology. Empirical evidence supported the hypothesis that students used Canvas® as a course management system; however, there were numerous responses which suggested that some faculty members have been resistant to, or have underutilized, the course management system. Based on the results from the questionnaire, and suggestions from operational staff at Florida Tech, this report concludes with the recommendation of specific human-centered design optimizations, which were evaluated by a team of graduate students.

 

 

 

Introduction

This report constitutes part of a team research project for the Organizational Management and Design course in the Human-Centered Design Institute. This report focuses on course management and teaching at Florida Tech primarily from a graduate students’ perspective. There are several concepts related to student achievement and learning outcomes. The report concludes with optimizations for a human-centered redesign of some key processes, events and activities that are related to course management and teaching at Florida Tech.

About Florida Tech

Florida Tech is a private, co-educational university in the USA. The Fall 2017 enrollment at Florida Tech’s main campus in Melbourne comprised 3,446 Bachelor’s students, 938 Master’s students and 561 Doctoral students (Office of Institutional Research 2017). Florida Tech is credited with having a personalized learning environment since the student to faculty ratio is 14:1 based on 4,665 students and 339 faculty (Office of Institutional Research 2016). In 2016, 80% of classes had less than 30 students (The College Board 2016). These advantages allowed students to work closely within teams and receive personalized attention from professors.

Course Management Systems

Students view courses through the University Catalog on web pages and in printed format. The Panther Access Web System (PAWS) enables students to register, add and drop courses via the web. Florida Tech has adopted Canvas® as its course management system (CMS). Canvas® provides a framework of software tools for students and faculty to interact with each other, course materials, assignments and grades. At Florida Tech, course delivery through Canvas® is not mandatory for faculty; however, students may benefit from consistent usage of the CMS. In spite of the advantages associated with CMSs, universities may face resistance to technology integration from faculty (Howard 2013; Jeffrey et al. 2014; Howard and Gigliotti 2016; Hulme and Winstone 2017). Florida Tech students access Canvas® with a TRACKS single sign-on account.

Teaching Practices & Student Engagement

At Florida Tech, instructors use a range of teaching practices which includes lectures, in-class activities, demonstrations, printed materials, videos and website references. Assessments generally take the form of assignments, labs, exams, quizzes, projects and presentations. In the research literature on higher education, there is consensus that student engagement has a strong positive correlation with student achievement (Zilvinskis et al. 2017; Kahu 2013; Oliver et al. 2008). This report sought to uncover which teaching practices were preferred by graduate students and were likely to boost engagement thereby positively impact their learning outcomes. The research hypothesis is that student engagement varies according to the compatibility between their preferences and the teaching and assessment methods selected by faculty.

 

Methodology

Informed Consent and IRB Requirements

Documentation for the Institutional Review Board (IRB) was prepared (IRB 2017). The research questionnaire was voluntary, anonymous and solely conducted with the informed consent from the student participant. The confidential research questionnaire had no risks associated with it, was specific to graduate students, and was relevant to staff involved in Graduate programs.

Anonymous Student Questionnaire

The goal of the anonymous student questionnaire was to discover reliable patterns related to course management and teaching at Florida Tech. The questionnaire incorporated engagement indicators from a modified and abridged version of the National Survey of Student Engagement (NSSE), which has been completed by approximately 6 million university students (NSSE 2017). The questionnaire was conducted using (1) face-to-face introductions where the researcher randomly approached students and asked if they were graduate students willing to participate in the research study, and (2) an online invitation in a WhatsApp channel and an OrgSync group where the researcher asked colleagues for assistance with the research study. If the student accepted to participate and agreed to the informed consent form, the researcher used a computer and web browser to share a readable hyperlink to a web-based form.

Interviews with Operational Staff

Four semi-structured interviews were conducted with the following Florida Tech staff:

• Jared Campbell (Instructional Technologist)
• Eric Donath (Canvas Administrator and Instructional Programmer/Analyst)
• Liz Fox (Director of Catalog and Graduate Information)
• Oscar Williams (Director of Technical Support Center)

At the beginning of each interview, the researcher was introduced as a graduate student who was conducting a research study as part of a coursework in Organizational Management and Design at the Human-Centered Design Institute of Florida Tech. Initial responses from the student questionnaire were used as a basis for formulating the interview questions.

Evaluation of Improvements

The results from the interviews were used to provide a preliminary validation of the improvements that were elicited from the questionnaire. Further, a sample of “expert” graduate students evaluated the final list of improvements using an importance rating scale. The sample of graduate students also provided suggestions related to the risks and feasibility challenges associated with the implementation of the proposed list of improvements.

 

Results

Responses from Student Questionnaire

The student questionnaire was conducted at Florida Tech for two weeks between 6 November, 2017 and 20 November, 2017. There were 26 responses from 21 (80.8%) males and 5 (19.2%) females who originated from Africa (3), Asia (10), Caribbean (5), Europe (2), North America (5) and South America (1). All students responded to the 15 questions that focused on student engagement, course satisfaction and technology habits.

In terms of Colleges and Departments, 21 (76.9%) were from the College of Engineering and Computing, 2 from the College of Science, 2 from the College of Business and 1 from the College of Psychology and Liberal Arts. The students were required to rate their level of experience using a personal computer on a scale of advanced, intermediate, novice and none. Of the 26 respondents, 15 (57.7%) rated themselves advanced, 10 (38.5%) as intermediate and only 1 (3.8%) person stated that they have no experience using a personal computer.

Results from Staff Interviews

The general results from the interviews are presented in the following Table 1.

Interviewee	Results
Jared Campbell Instructional Technologist	Canvas® adoption challenges Course management technologies CMS improvement plans
Eric Donath Canvas Administrator & Instructional Programmer/Analyst	Canvas® gamification concepts Canvas®-related support requests Canvas® support channels
Liz Fox Director of Catalog & Graduate Information	Courses offered at Florida Tech Digital catalog system called aCalog Policy management system
Oscar Williams Director of Technical Support Center	Technology Support Center requests

 

Table 1 – Results from Staff Interviews

 

Findings

Likert scales used in the questionnaire were presented using ordinal labels with associated numerical weights. In the analysis, weights for each option were summed and computed as percentages of the maximum possible sum. These percentages were used to rank the options.

Student Satisfaction with Courses

The ratings of satisfaction with courses were positively skewed with 73% in agreement and only 7.7% expressing dissatisfaction. There were 19.2% of the students who maintained a neutral position on satisfaction with their courses.

Student Engagement Indicators

 

The 10 student engagement indicators ranked by their percentage weights were as follows:

1. Quality of Interactions (89%)
2. Learning Strategies (88%)
3. Effective Teaching Practices (86%)
4. Student-Faculty Interaction (85%)
5. Higher Order Learning (85%)
6. Collaborative Learning (85%)
7. Discussions with Diverse Others (83%)
8. Reflective & Integrative Learning (83%)
9. Quantitative Reasoning (82%)
10. Supportive Environment (82%)

Students generally felt that high quality interactions and supportive relationships were most engaging. The highly rated personal learning strategies used by students implied that students were actively engaging with course material. Effective teaching practices and experiences with faculty members were not surprisingly rated very high. Additionally, higher order learning was rated on par with collaborative learning as more engaging than quantitative reasoning, which implies that students were more likely to be engaged with evaluation, synthesis, analysis, and interpretation rather than simply absorbing knowledge facts, names and numbers.

Effective Teaching Practices

 

The 12 selected teaching practices ranked by their percentage weights were as follows:

1. Projects (88%)
2. Assignments (84%)
3. In-class activities (82%)
4. Lectures (82%)
5. In-class demonstrations (80%)
6. Video tutorials (80%)
7. Websites (79%)
8. Online knowledge base (79%)
9. Learning management system (75%)
10. Exams (75%)
11. Printed materials (71%)
12. Quizzes (70%)

Analysis of teaching practices from the students’ perspective revealed the effectiveness of practical “knowledge pull” methods (projects, assignments, in-class activities) superseded the more passive “knowledge push” methods (lectures, in-class demonstrations and video tutorials).

Technology Contribution to Courses

Students confirmed that technology contributed significantly to their learning, studying, or completing coursework on their own as well as with other students . Even though this finding related to technology was positive, 100% of the students admitted that technology distracted them to some extent from completing their coursework. 57% of the students stated that this distraction was quite a bit (19%) and very much (38%). This finding is of concern and should be investigated further in order to moderate the adverse impact of technology on coursework.

Frequency of Technology Usage

 

In order to understand how often students were using technology, 8 different technologies were surveyed. The technologies ranked by their percentage weights were as follows:

1. Mobile computing (92%)
2. Course management tools (89%)
3. Collaborative editing software (83%)
4. Electronic textbooks (83%)
5. Multimedia software (82%)
6. Social networking (82%)
7. Online portfolios (72%)
8. Blogs (67%)

Mobile computing dominated the range of technology used. In spite of the fact that social networking ranked number 6, there were 73% of the students that used social networking often (23%) and very often (50%). Three (3) learning related technologies ranked highly within the top 4 most frequently used technologies, namely course management tools such as Canvas®, collaborative editing software such as Google Docs, and electronic textbooks. 23% of the students were unfamiliar with blogs and 12% were unfamiliar with online portfolios.

Course Management Technologies

According to Jared Campbell, Eric Donath and Liz Fox, Canvas® interfaces with other technologies in the overall course administration system. For example, courses that are managed in Canvas® may be cross-referenced from Banner®, which is a relational database used by Florida Tech for administration and enterprise resource planning. Other third-party technologies commonly used with Canvas® are listed as follows:

• Panopto™ – for recording lectures, screencasts and video content management
• Poll Everywhere® – for live interactive classroom participation and student responses
• Turnitin® – for student submissions and automatically checking for plagiarism

Mrs. Fox described the university’s endeavor to synchronize course catalog content and various policies through a single source. The administration of syllabi was a common theme in the interviews with Mr. Campbell and Mrs. Fox. Both of the staff were actively implementing methods to make the process more faculty-led and faculty-friendly. The idea was to have faculty instructors easily generate their syllabi and upload them on Canvas®.

Features on Canvas® used by Students

The questionnaire revealed that 96% of the students used Canvas® for course management. The features used by these students were ranked as follows:

1. Grades (92%)
2. Assignments (88%)
3. Files (88%)
4. Announcements (76%)
5. Syllabus (68%)
6. People (64%)
7. Quizzes (48%)
8. Multimedia (44%)
9. Discussions (36%)
10. Chat (32%)
11. Collaborations (24%)
12. Conferences (16%)

It can be determined that the top 5 features used were information oriented. The communication oriented features of Canvas® were less commonly used e.g. discussions (36%), chat (32%), collaborations (24%) and conferences (16%). It is notable that the most commonly used Canvas® features were related to assessment i.e. grades and assignments.

Even though communication via Canvas® was not high, it was outstanding how much the students used technology to communicate with their peers, academic advisors, faculty and to a lesser extent with staff at student services and administration.

Student Challenges with Canvas®

From the students surveyed, 60% experienced no problems using Canvas®. In spite of the frequent usage of the CMS, some students reported common challenges. Most notably, 56% of the students reported human-related challenges i.e. 40% claimed that not all of their professors used the CMS effectively; 8% reported that their professors did not use the CMS at all; and, 8% reported that they had limited access to tech support related to Canvas®. Some of the other challenges reported included the following:

• The CMS was unavailable when they needed it
• The communication features were unreliable
• Some software applications did not work properly with the CMS

Staff Perspectives on CMSs

The interviews with Jared Campbell and Eric Donath revealed that the Office of Information Technology at Florida Tech was making a concerted effort to have the Canvas® CMS adopted by all students and faculty. Mr. Campbell reported that there is no mandatory policy for the use of Canvas® even though it was recommended to faculty.

Mr. Donath suggested that gamification can be used proactively to induce greater adoption. The proposed gamification strategy involved benchmark ratings of professors within colleges and departments as a means to motivate department heads and deans to become advocates for promoting usage of the CMS among the professors.

To supplement this strategy, a basic 8-module training course called “Canvas® 101” has been deployed and made available to professors. To date, Mr. Campbell reported that approximately 173 of 339 faculty members (51%) have completed the training course on their own volition.

Additionally, the instructional technologists have developed resources such as syllabus templates and interactive web forms to assist professors with the development of well formatted, standardized syllabi. If this process becomes adopted as a best practice, students reported that they will be more satisfied, especially if the syllabus is located in Canvas® on a separate tab.

Mr. Donath stated that there were 1,907 Canvas® support requests from students and faculty between Sept 5, 2016 and Oct 9, 2017. There were approximately 200 Canvas®-related requests for administrative tasks e.g. connecting third-party apps. Canvas®-related requests can be reported to Canvas® Support 24/7 via phone, email, online support request or live chat. These support channels were accessible from the Help link in Canvas.

Mr. Oscar Williams stated that there were 13,869 student support tickets between Sept, 2016 and Oct, 2017, and 9,479 faculty/staff support tickets on the same time period. Therefore, the percentage of Canvas®-related support tickets was approximately 8.2%. This ratio supports the strategy of the Information Technology department to create Canvas®-dedicated support channels that would separate the requests from the list of general technology support requests.

The interview with Liz Fox provided insights into the process by which courses were entered into the catalog, and the challenges with synchronization of course information, syllabi and administrative policies. Ms. Fox explained the university’s advancement to supplement the printed and portable document format (PDF) catalogs with Acalog™, which is Florida Tech’s online catalog management system.

Based on information provided by Mrs. Fox, there were approximately 2,541 courses in the catalog of which 1,244 (49%) are numbered level 5000 or higher (i.e. a general indicator of courses that are intended for graduate level). Each course has a specific description, list of pre-requisite courses and syllabus. Synchronizing up-to-date information, especially syllabi in a common format posed a recurring challenge to the Registrar’s Office.

 

Proposed Optimizations

The participants of the questionnaire shared their ideas for improvement of the processes, events and activities at Florida Tech. These ideas were categorized into concepts which were rated by the team of graduate students. The following list of ranked optimizations emerged.

 

1. Real-World Learning

There should be greater emphasis on practical topics that directly apply to industry. Graduate students do not only wish to be taught but they demand learning through critical thinking and working on relevant projects and applications. Internships, co-ops and on-the-job experiences were highlighted to ensure the transfer of knowledge into industry.

2. Collaborative Learning

Related to real-world learning, teamwork and collaboration were highly important aspects of work readiness. To this end, teaching methods should comprise more collaborative projects and in-class activities that feature high order learning and peer discussion, which develop skills related to the analysis, synthesis, evaluation and creation of knowledge.

3. Gamified Assessment

Consistent practise and student engagement are imperative for continuous learning. For Millennials and Generation Xers, gamified assessments may be effective replacements of traditional homework and paper-based quizzes. These games provide prompt feedback and blend powerful learning media to reinforce difficult concepts. If these gamified assessments have minor but positive impact on grades, students may be more receptive to consume them.

4. Personalized Planning

Currently Acalog™ facilitates searching for courses and printing or downloading degree plans.
An upgrade that makes it a personalized, interactive program planner with real-time grade monitoring is proposed. By allowing students to review their cumulative progress in real-time and simulate the impact of future grades may motivate better student performance.

5. Canvas Support Package

There are Canvas® tutorials distributed all over the Internet e.g. https://vimeo.com/74677642. There is also the “Canvas 101” course that is available through the Instructional Technology department. However, students and faculty may be unaware of these support resources that are available to help them become better users. The research proposes a well promoted set of scenario-based micro-webinars and videos for students and faculty to become familiar with the CMS. Supplementary assets such as templates for syllabi and model courses may help faculty to onboard with the CMS and meet the demand of students for using the platform.

 

Conclusion

In this report, course management and teaching at Florida Tech was investigated using conventional surveying methods based on paradigms from higher education. The questionnaire used was proven as a reliable measure to elicit the trends and perceptions of students in relation to their preferences. The constructs used were rooted in operational definitions espoused by the National Survey of Student Engagement (NSSE 2017). The measures of student engagement, learning outcomes and effective teaching practices yielded remarkable results and findings upon which the proposed optimizations were founded.

Graduate students favored interactions with faculty and peers highly. They prefered a deeper understanding of the information that had been communicated by their professors and appeared to be keen on using technology in their learning and studying. There was significant evidence that Canvas® was being used by the students; however, the major challenges related to the adoption of the course management system resided on the side of the faculty. Donald Ely, in his seminal research of technology diffusion, identified eight main conditions that facilitate the implementation of educational technological innovations, which are relevant to Florida Tech (Ely 1990; Ely 1999). The eight conditions can be used to add structure to the implementation of the proposed optimizations.

The operational staff at Florida Tech who are responsible for course management were open to the research project. Going forward, this research suggests that they should consider the faculty as a key persona and discover more about them in relation to their (1) dissatisfaction with the status quo, (2) existence of knowledge and skills, (3) availability of resources, (4) availability of time, (5) existence of rewards or incentives, (6) participation, (7) commitment, and (8) leadership (Ely 1990; Ely 1999; Anderson 2008). The RIPPLES model has been used in higher education to successfully integrate technology, and to streamline the readiness for CMS implementation in the areas of Resources, Infrastructure, People, Policies, Learning, Evaluation and Support (Tomei, 2008).

The final list of optimizations proposed are student-centered, practical in nature, and have no substantial risks associated with them. They will, however, require buy-in from the highest administrative level to the department head level all the way down to the faculty level. With commitment, Florida Tech can enhance its course management and teaching functions in accordance with the university’s mantra – “High tech with a human touch”.

 

References

Anderson, C. (2008). Barriers and enabling factors in online teaching. International Journal of Learning, 14(12), 241-246.

Ely, D. P. (1990). Conditions that facilitate the implementation of educational technology innovations. Journal of Research on Computing in Education, 23(2), 298-305.

Ely, D. P. (1999). New perspectives on the implementation of educational technology
innovations. (ERIC Document Reproduction Service No. ED4277775).

Howard, S. K. (2013). Risk-aversion: understanding teachers’ resistance to technology integration. Technology Pedagogy and Education, 22(3), 357-372.

Howard, S. K., & Gigliotti, A. (2016). Having a go: Looking at teachers’ experience of risk-taking in technology integration. Education and Information Technologies, 21(5), 1351-1366.

Hulme, J. A., & Winstone, N. E. (2017). Do no harm: Risk aversion versus risk management in the context of pedagogic frailty. Knowledge Management & E-Learning, 9(3), 261–274.

Institutional Review Board. (2017). Institutional Review Board (IRB) at Florida Tech. Retrieved from http://web2.fit.edu/crm/irb/.

Jeffrey, L., Milne, J., Suddaby, G. & Higgins, A. (2014). Blended Learning: How Teachers Balance the Blend of Online and Classroom Components. Journal of Information Technology Education: Research, 13, 21-140.

Kahu, E. (2013). Framing student engagement in higher education. Studies in Higher Education, (38)5, 758-773.

National Survey of Student Engagement. (2017). About NSSE. Bloomington, IN: Indiana University Center for Postsecondary Research. Retrieved from http://nsse.indiana.edu/html/about.cfm.

Office of Institutional Research. (2016). Common Data Set. Retrieved from http://web2.fit.edu/oir/documents/doc_mgr/1192/CDS_2016-2017.pdf.

Office of Institutional Research. (2017). Student Enrollment at Florida Tech. Retrieved from http://web2.fit.edu/oir/student-enrollment.php.

Oliver, B., Tucker, B., Gupta, R. & Yeo, S. (2008). eVALUate: an evaluation instrument for measuring students’ perceptions of their engagement and learning outcomes. Assessment & Evaluation in Higher Education, 33(6), 619-630.

The College Board. (2016). Annual Survey of Colleges. Retrieved from https://www.collegedata.com/cs/data/college/college_pg04_tmpl.jhtml?schoolId=882.

Tomei, L. A. (2008). Encyclopedia of Information Technology Curriculum Integration (2 Volumes) (pp. 1-1146). Hershey, PA: IGI Global. doi:10.4018/978-1-59904-881-9

Rhodes, T. & Finley, A. (2013). Using the VALUE Rubrics for Improvement of Learning and Authentic Assessment. Washington, DC: Association of American Colleges and Universities.

Zilvinskis, J., Masseria, A. & Pike, G. (2017). Student Engagement and Student Learning: Examining the Convergent and Discriminant Validity of the Revised National Survey of Student Engagement. Research in Higher Education, 58(8), 880-903.