Forum Analysis

	Numeric Analysis of 117,000 Messages Simple Analysis of the Message of an On-line Community
	Abstract: Analysis of four Lotus Notes databases that contain the messages of a growing community over a nearly seven-year period offers interesting and useful insights. We looked at the growth in gross number of messages, the changing population of members and how those can be categorized, and we looked at various classes of the messages to see what we could learn.

Short Version of Evolution

So, a discussion database was set up, and partners notified they could connect their servers to Notes Net, and communicate. And the BP stayed away in droves. By erecting the discussion, and effectively leaving it to the participants to decide how to use it, Lotus discovered that few people had the motivations or the interest. And those who did visit for the first time were disappointed in what they saw, and never returned. The results were pitiful.

It looked like a promising start in May '91, with a progression of growing number of posts per month. But without any intrinsic "draw," something to really bring people into the cyberspace and keep them there until they begin to communicate among themselves, participation dwindled. Then, in October, '92, the numbers really started to climb again.

	Without attention and intention, discussion databases languish; this one shows a period of eleven months during which there were only seven posts. There's too little content to attract people to visit frequently, and without lots of people there's never a "critical mass" of core frequent participants to ensure a welcoming space to future newcomers. Getting to and sustaining critical mass is a critical success factor for any discussion database. But, something magical happened in September, '92, which inspired more people to participate in October and November. And the growth continued from there. Nobody seems to know what happened. Perhaps more BP had begun to get connected to the world; perhaps Lotus' BP Program department publicized the availability. Nobody recorded why, but it commenced spontaneous growth.		It's the deadly embrace that kills most on-line discussions: Nobody posts, so there's nothing to read, so nobody posts.
	Then and Now Analysis of the BP Technical Forum shows an astounding growth: It took more than 28 months to accumulate 3,000 posts; during 1997, every month that number was exceeded.

A pattern of consistent growth is evident, despite variations. The four distinct colors represent four successive database designs. You can also see the transition times, during which participants had to monitor both an "old" and a "new" forum. The transitions spanned four months; evidence suggests we've learned how to improve that; the February, 1998 transition (after this data) has happened in one month

Post Volume Growth, Five years, by Month

The four distinct incarnations of the "Partner Forum" each had different names. Originally, the goal was to enable communication among Lotus Business Partners, without much regard to content or focus. Hence, the generic name. When the Partner Forum grew to nearly 20,000 messages, it was supplanted by the "New Partner Forum 10/94+" (October '94 and onward). There was some significant transition confusion at that time; the older database was disabled prematurely, significantly affecting the growth.

By early '96, the need for a new database was evident. There were lots of wishes for changes in design, and we then past 25,000 posts. With many users clamoring for a new database, but many unresolved issues about design on the table, Lotus spawned a new discussion, called the "Forum Design Discussion." On his own initiative, Alex Wilson, one of the more prolific members of the community, designed a whole new database and circulated it to willing peers for experimentation.

There's a sad tale of how the "BP Technical Forum 96" came to be deployed, and Alex Wilson was very angry as a result. The community came together to express outrage that one of their colleagues should be so shabbily treated, yet it was a blessing in disguise: It allowed the nascent community to express a unified voice in support of controlling their own destiny. Eventually, the entire brouhaha settled down. And, in the process, one of the changes made was to rename the database "BP Technical Forum 96", forever raising the technical bent of the forum to the stature of label and identity.

All of us having learned our lessons, Mike Woolsey, who'd been consistently active since early '93, stepped into the space vacated by a bruised and abused Alex Wilson, and created yet another iteration in design excellence. That design was deployed in June, 1997, as "BP Technical Forum 97." Wilson's design was so sound, Woolsey's improvements so significant, and key Lotus employees had become so much more collaborative, the design was merely fine tuned and became the basis of "BP Technical Forum 98" in February (which is why our analysis in this monograph stops in January, 1998).

To this day, the design continues to evolve (the latest tweaks were a mere two weeks prior to this writing). Because the designers and the people who advise on the design are active community participants, it is one of the richest and convenient discussion I've ever used. It is a sterling example of what collaboration and hard work by stellar designers can accomplish. Compared to it's vibrant colors, other Notes discussion databases I've used are mottled pastels. And, comparing the Notes discussion database we use in the Forum is not unlike contrasting fine art and childlike scribbles.

Throughout this monograph, I use "The Forum" to denote the totality of all the various discrete designs that were deployed. Where it is important to distinguish between the discrete designs, I use their specific names:

Forum Title	First Post	Final Post
Partner Forum	May '91	Dec. '94
New Partner Forum, 10/94+	July '94	Sept. '96
BP Technical Forum 96	Apr. '96	Sept. '97
BP Technical Forum 97	June '97	Feb. '98

Even disasters may
become opportunities
for community
formation.

Corps of Regulars

Every community thrives on continuity: Some members who are there before newcomers arrive, and are there long after the casual participants depart. In most cases, Pareto's Rule of 80/20 applies, and it's no exception here. The Partner Forum had a cadre of about 20% who have participated for six or more consecutive months of at least one post each month^¹. And, as one Regular move on to other pastures, there's inevitably someone else who steps into the gap. A few members of the community have participated in the various incarnations of the Partner Forum for five years or more. They carry (and shape) the culture of the cyberspace.

The Core Regulars carry the culture of the community across time, exhibiting behaviors that New participants can model. Most participants are neither Regulars or New, but casuals, who may drop by for a specific solution, or to offer some additional experience.

(The apparent decline in final months is an artifact of the rolling average method used to categorize Regular and New participants.)

We learn to read before we write; the effective participant in a discussion database reads posts to observe the customs of the cyberplace before they post. Whether they post long or short messages, or they include attachments or not, or whether they use hostile or abusive language or not is determined for most people by the volumes of messages on file that newcomer reads. And the exemplary practices of the cybercommunity are most often shaped by the Regulars. The newcomers and casual participants have models to emulate in style.

There have been about 100 Regulars active in the Forum for the past year; they comprise about 20-25% of the total number of unique people who participant, month-in and month-out. Newcomers (defined^² as people who consistently post for three or four months, never having been seen before); account for about 10% of each months' participants. Everyone else is part of the General population, those who infrequently participate, or stop by then move on.

About a quarter
of your members
will be Regulars;
court them.

Patterns of Messages

It's interesting exploring the pattern of messages over time. In looking over more than 100,000 messages (I've made no attempt to re-read them all), there are some patterns that are rather obvious.

Questions & Answers

The core of the Forum is Q&A about issues that are important to the originator of the thread. Often, the responses are proclaimed richer, more relevant or faster than answers that might be expected from the traditional sources, such as Lotus' own Technical Support. However, part of the usage guidelines for the Forum make it clear that it is not to be considered a TechSupport alternative.

It's not surprising that the Forum surpasses TechSupport in some cases: The Forum is populated with practicing professionals, some of whom have years of experience in delivering solutions to customers. Traditionally, TechSupport personnel are highly trained in the specifics of the product, but may not yet have spent much time in solving customer's problems.

However, focusing on these difference in experience levels hides even more important distinctions between the Forum and Lotus' TechSupport:

	Partner Forum	Lotus TechSupport
Certainty of Response	No	Yes
Experience-based solutions	Yes	Seldom
Escalation processes for follow-thru on complex issues	Limited	Yes
Access to developers, other internal Lotus expertise	Occasionally	Yes
Opinions and other "fuzzy" information	Yes	Seldom
Quick response (<24 hrs)	Seldom	Yes

Clearly, there is a place for both the informal Forum, and the more formal capabilities and processes of the formal TechSupport system. Often, when a new participant begins to treat the Forum as an alternative to TechSupport, more experienced people chide them for not using the appropriate resources.

Wishlists / Feature Requests

From earliest days of the Forum, participants posted wishes and dreams for what they would like to see in the Notes/Domino product that would make it easier to deliver solutions to customers. During the first four years, few of these requests were formally acknowledged by Lotus. However, from April, 1996, when Scott Brown was assigned formal authority to act as "go-between" for Forum members and Lotus, he (and, subsequently, his colleagues) began forwarding "wishlists" and feature requests to the cognizant Lotus employees. And, when the "BP Technical Forum 96" was deployed, it contained a new document type named "Wishlist", which provided a means to make sure relevant information was appropriately tagged. And, a "Vote" type was provided for other participants to add their support (or lack thereof) for the idea, so Lotus would have some gauge as to the importance of the issue to these Business Partners.

Wishlist messages have now become a formal part of the feedback Forum members can give to Lotus. And now, instead of the idea just sitting there for Lotus' consideration, Scott and his team post a message when they've transferred the messages into the developer's internal system for enhancement and bug fix requests. And, as part of community building between Forum members and Lotus, we often see "Thank you" messages appended to the original thread then a new release contains the enhancement requested.

Problem and Bug Reports

Because so many of the Forum members are operating at the leading edge of customer solutions, they often come across anomalies or subtle bugs that are virtually inevitable in a product as complex as Domino and Notes. However, these sometimes have also been evidence of a lack of understanding of the product or newcomers' inexperience with the product. In the earliest days of the Forum, participants might post a "Bug Report," but even if true, there was no certainty that the report would be seen by Lotus' developers. Message authors were encouraged to also report the problem through Lotus' Technical Support to make sure the report was entered into the company's internal problem report databases for future consideration.

Like "Wishlists," the handling of "Probable Bug Reports" (PBR) have evolved. The reasons PBRs are encouraged in the Forum are several:

For rare or subtle bugs, other Forum participants may be able to shed light on the conditions by offering their own experience or additional sample cases because they now know they're not the only people experience the issue,

When people post "Bug Reports," the reason they're called "Probable" is that sometimes what perceived as a "bug" is, in fact, the intentional way the product works. Fixing that problem might prohibit some other essential feature. The Forum members, with their rich variety and depth of experience, can often help the reporting member understand the issue, or offer temporary workarounds. Thus, extraneous bug reports never make their way into Lotus' internal reporting systems.

During 1997, Carolynn Rago joined Scott Brown's team, and has distinguished herself by actually setting up the reported configuration in an attempt to duplicate the problem. If she successfully reproduces the problem, she reports in the forum the "case number" assigned to the problem. In many cases, she finds that the problem has been previously reported, and can another case example to the file.

If the problem cannot be duplicated in the laboratory, Carolynn reports that to the Forum, providing the opportunity for the original message author to expand on the conditions. Sometimes, other members add test case data that help identify the specific conditions that induce the problem, resulting in a formal problem report being generated within Lotus.

Social Lubricants

Many of the messages...probably as many as 25%...are what I call the "social lubricants." These are "thank you" and "good idea" and "Yes, me too" posts that serve to reinforce the desirable behavior of the member who authored the original post. As I've pointed out earlier, these posts are part of the community ambience.

Some of these messages are, frankly, irrelevant to the main topic of the database: A recent thread about the release of a new version of the Quake computer game, exchanges of jokes or stories. Unless and until some participants begin to complain about the volume of such "off-topic" posts, they tend to be tolerated. In most cases, the participants themselves allow the discussions to dwindle out without complaints from other members.

This is the pattern
of messages in
the Forum.
Each community is
bound to be unique.

Evoking Passion

There are two major forms of discussion fora, called Bulletin Board and Outline.

Bulletin Board style is the simplest; the technology provides no tools for capturing the connection of original posts and their dependent responses. Each new post is added to the collection in simple chronological order; it is the responsibility of the author of each individual post to provide guidance as to which prior messages this response may relate. The Internet USENET newsgroups are discussions of this form, and each message typically includes some (or all) of an earlier message to provide context and linkage.

It is hard to isolate the specific "threads" of discussion in a Bulletin Board style of discussion. At best, one can measure along two dimensions: the number of posts in a discussion, and the period over which they occurred.

Outline style maintains, through indenting of lines, a "post/response" relationship. One message is a principal post; other messages related to that original are displayed under, and indented toward the right. And responses may have further responses. The hierarchy of original/response relationship is maintained, even at the expense of keeping things in chronological order. These discussions are often call "threaded," as the relationship of multiple (often simultaneous) threads of the on-going discussions are preserved.

It is much easier to isolate which posts "belong" to the same discussion in Outline form, because they're naturally isolated. An Outline form discussion can be measured along three dimensions: The total number of posts in the entire collection, the number of levels of indenture to which the discussion is taken, and the period of time spanned from the first message to the last. The level of indenting suggests the complexity of the discussion. The time span (usually measured in days) usually relates to the importance of the issue to the participants.

All discussions in the Forum are of the Outline form. It is a native property of Lotus Notes.

The most lively discussions are those that have the largest number of messages, the deepest number of indent levels, for the longest period of time. In the Forum, the most lively and enduring discussions have generally been about non-technical issues. The largest discussion of all (416 messages, 21 levels deep) was about the rules for governing the Forum itself, in mid-1996.

From inception, the Forum has experienced 19 threads with 100 or more messages. One of those threads was a response to an offer by Randel Oulton for a free Notes database full of icons. Every one of the other eighteen had some contribution to make toward the formation of community:

#	Level	Days	Start Date	Class	Summary
416	21	117	2/24/96	Gov	Debate over proposed rules
182	15	154	1/25/94	Rel	"Where does everybody live?"
152	16	54	1/21/94	Val	About "guys" and gender-specificity
132	6	220	5/17/95		(Requests for free icon database)
125	13	36	12/15/93	Rel	Remembering early computers
124	21	66	10/19/93	Gov	Appropriateness of humor in Forum
124	8	31	3/31/95	Isu	Unhappiness with billing procedures
120	11	207	1/30/95	Isu	Unhappiness with certification exams
117	11	203	9/4/93	Gov	E-mail etiquette and legalities
117	5	53	2/4/96	Rel	"Happy Birthday" in many languages
116	3	89	5/18/94	Gov	Appropriateness of large documents
115	17	427	10/11/93	Isu	Opinions about User Interface design
105	12	91	5/16/96	Isu	Product reliability experiences
105	4	112	6/10/94	Rel	Privacy/publicity of member info
104	20	37	2/22/94	Isu	Encouraging change in Lotus policy
104	12	7	10/31/96	Rel	Long thread about being a long thread
103	8	17	1/5/98	Isu	Benefits of becoming certified
102	18	558	3/11/93	Gov	Rules for governing growing Forum
101	7	90	6/28/95	Isu	Brainstorming TV ads for Lotus

Legend

# Total number of posts in thread

Level Number of indent levels

Days Period of discussion

Start Date Date of original post

Class Relevance to community formation:

Code Explanation

Gov About governance and rules

Rel About each other, and relationships

Val About values shared (and not shared)

Isu About issues for which strong opinions (both pro and con) are expressed

Summary Summary description of thread contents

These eighteen threads, encompassing over 2,400 messages, represent significant moments in the formation of community. It was in these ostensibly off-topic, "fluff," or "time-wasting" dialogues that much of the work of community values was threshed out. The specific topics, in many cases, were utterly irrelevant: A long discussion about the use of "guys," and how some women feel excluded (and many "guys" feel about that) could have been about another other emotion-laden topic. It exposed styles, and values and communication skills of the participants. It allowed people the space (or cyberspace) to get to know each other.

By way of contrast, at the other end of the spectrum, there were nearly 2,700 original main topics posted that never garnered a single response. And, most of these were requests for technical help or posed other technical questions.

Participants invest
their energies in
highly-charged
debates and issues.

Footnotes:

^¹

"Regular" criteria: A person who has posted at least four messages per month for at least seven out of nine months. The labeling as "Regular" is computed for each month by examining the current month, the preceding four months, and the subsequent four months.

^²

"Newcomer" criteria: A person who, never having posted before, posts at least two messages per month for three months. After a maximum of three months, a 'Newcomer" becomes a Regular by meeting those criteria, or reverts to being called a member of the General population.