Last year, a colleague shared with me how he used ChatGPT to evaluate content generated by his students at a local university. I was intrigued by the idea but also concerned about the ethical issues surrounding using publicly available LLMs to read student writing without their consent. This led me to look for locally hosted LLMs that could aid in grading and evaluating student work without putting their work in a publicly accessible large language model.
I am not qualified to evaluate writing, but I do use Grammarly to help me with my writing. Are there open-source tools in addition to proprietary products like Grammarly that could aid me in the writing process, and could they be successfully used by university staff and students to improve writing outcomes?
My experience with Ollama led me to consider using it to evaluate my writing and suggest to others that it could be used to assess student writing. If you haven’t already done so, you will need to install Ollama. The commands below will install it on Linux, which is my daily driver. There are detailed instructions on the Ollama website to help you download and install it on MacOS and Windows.
curl -fsSL https://ollama.com/install.sh | shI decided to use the Llama3.1 model for this project. I issued the following series of commands in the terminal to launch Ollama, download the Llama3.1 model and then run that model.
$ ollama pull llama3.1
$ ollama run Llama3.1
>>>Evaluate the following writing sample,"
I copied and pasted an article I had written into the Ollama REPL (Read Evaluate Print Loop) and pressed Enter. The fourteen-hundred-word article was written in 2016 about a lecture our Franciscan Fraternity had attended on the similarities between our faith and Islam. The evaluation took several minutes on my powerful Linux desktop, which has a quad-core 11th Gen Intel Core i7-1165G7 with 64 gigabytes of RAM but lacks a GPU, which is very important when using Ollama and these models. Nonetheless, it completed the task and provided a three-hundred-fifty-five-word writing analysis. A sample of the analysis is provided below.
“This writing sample appears to be a reflective essay or report on a lecture by Fr. Michael Calabria about Islam from a Catholic perspective. Here’s an evaluation:
Strengths:
Clear structure: The text is well-organized, with a logical flow of ideas and a clear conclusion.
Engaging narrative: The writer effectively conveys the content of the lecture, making it easy to follow and understand.
The analysis provided a list of weaknesses and suggestions for improvement.
“Weaknesses:
Lack of a clear thesis statement: The writer’s purpose or main takeaway from the lecture is not explicitly stated, leaving the reader to infer the significance of the event.
Lengthy and detailed report: While the narrative is engaging, the text feels overly long and detailed, which may make it challenging for some readers.
Suggestions for improvement:
Edit for clarity and consistency: Review the text for minor errors in formatting, punctuation, and grammar to ensure a polished final product.
Consider adding more analysis or reflection: While the narrative is engaging, some readers may appreciate more analysis or reflection on the event’s significance or what it means for the writer’s understanding of Islam.
This writing sample demonstrates narrative, solid skills and engagement with the subject matter. With some editing and refinement, it could become an even more substantial piece of writing.
Have you used Ollama or other open-source artificial intelligence to evaluate your writing? My next article will be about using Ollama and Llama3.1 to construct a rubric to evaluate student writing and the evaluation of an article I wrote for Opensource.com, 12 Essential Linux Commands for Beginners. All output from Ollama is provided in MarkDown format. What implications does this have for your writing and for the evaluation of student writing?