Pretotype 3 - Chatbot to search for open-source projects in GitHub

Idea

Bot that finds open-source tools, libraries and frameworks using GitHub.

Market Engagement Hypothesis

Many people don’t use open-source software because they don’t know where to find them or how to effectively search for them, so they rely on commercial or proprietary software. This is especially true because open-source software has so many options that, without spending some time searching, the first options someone tries might not be what they are looking for and cause them to give up.

XYZ Hypothesis

X – At least 30%

Y – Encora developers

Z – Will use the bot to search for an open-source project once a week.

At least 40% of Encora developers will use the bot to search for an open-source project once a week.

Experiment

Take a subset of developers, in this case, Apprentice colleagues and mentors, to try out a simulation.

XYZ Hypotheses

1 – At least 50% of invited participants will use the bot once during the experiment.

2 - At least 70% of active participants will receive a satisfying or useful answer.

Method

For this experiment, the mechanical turk method was used to simulate the bot’s functionality using a Slack channel where participants could write their requests and the researcher would answer them by manually searching the queries in the GitHub platform. Additionally, the one-night stand method was used, because the experiment duration was limited (just a couple of days).

Most invited participants used the bot at least once during the experiment, which shows interest in this tool.

Storyboard

The interaction was as follows: A Slack channel was created where participants could voluntarily write queries for the bot to answer, following guidelines previously defined, and a researcher would manually answer their queries by searching on the GitHub platform. For each query, two or three options were provided in the answer.

Resources

The necessary resources for this experiment were: a computer, Internet access and invited people participation. This means that, in reality, no expense was made, because the resources were already available to everyone.

The time spent on this experiment was 6 hours, including the Slack channel creation, the welcome and feedback questionnaires, and replying the queries.

Results

Initial interest in the tool was high. Almost 80% of participants admitted using open-source tools frequently, it is worth mentioning that 50% of these said that it depends on the program. On the other side, 60% of participants mentioned as limitation of open-source software the lack of technical support and unknown options, and this last problem is what this experiment tries to address. Compared to the initial hypotheses:

1 – 70% of the invited people participated in the experiment, a 20% increase from the original expectation.

2 – 100% of participants mentioned that the results were useful or satisfying, a 30% increase compared to the original hypothesis and a very positive number for the goals of this experiment.

In addition, 85% of participants accept that the tool would be useful for their jobs.

However, it is worth nothing that, despite the surprisingly good accuracy, the feedback questionnaire shows that 33% of participants mentioned that they would use the bot less than once a week; it is a high number but it is still within the original expectations.

This experiment shows the difference between ILI (Initial Level of Interest) and the real-life use. The preliminary results show that, despite the high level of ILI, real-life use will be modest. It can be useful to replicate the experiment using a larger number of participants to obtain more insight, but if a conclusion must be reached using the data shown here it would be that the tool is well-received and ready for a prototype.