[Apologies for crossposting but I think there enough different types of readers/users in these groups to give me a variety of opinions]
I need to design a "User's Forum" for an upcoming product release. The goal (as always) is to let users share their ideas, solutions, gripes, etc. with each other directly without going "through" any business structure.
I've been trying to put together an appropriate set of design criteria for that mechanism. The folks using this will be of varying degrees of "computer savvy" -- from "none" to "expert". The experience should be as friendly/annoying to users at either end of the spectrum.
A few issues are not negotiable; some have "very high inertia" (but could be "moved" given the right impetus) and others are just "whims".
I've pretty much decided that this will all be SMTP driven. This gives tighter control over "membership" as well as being damn near ubiquitous. It also means the service needn't be as "exposed" to potential hackery.
Since it is too easy for folks to harvest email addresses, I think the software should hide email addresses completely. (I may also suggest creating arbitrary "handles" for users so that their real names are, by default, hidden -- unless they opt to disclose them deliberately).
This poses a problem for "private" conversation. I've not decided how to handle that (e.g., provide a separate channel explicitly for this? But, I don't want to encourage it's use :< ).
I suspect a "file area" may be necessary -- otherwise everyone gets a copy of *every* file that *anyone* opts to post. :< But, that poses a problem for liability issues as well as making demands on how *much* can be "stored" there and by who...
The biggest issue is moderation. I don't want to have to have *a* moderator intervene in all posts. Yet, I want to ensure that the forum remains on topic and doesn't just become a haven for whiners needing someone to commiserate with, etc. I've thought of things like the Craig's List model of "self moderation" but I don't think that works in a "push" technology. I would hate to have to implement an HTTP/NNTP-ish server
*tickled* with SMTP pushes followed up with HTTP/NNTP "pulls-of-interest".
Are there other reasons that using an off-the-shelf solution, such as pgpBB or vBulletin wouldn't work in your application? I think the concerns you mention are valid, but I've dealt with several companies that use the off-the-shelf software for their user forums and it seems to work pretty well for them. Membership can be completely subject to approval if you so choose. It seems that not taking advantage of the benefits such software offers vs. some hypothetical security advantages of a SMTP driven system might be cutting off your nose to spite your face.
Why don't you just use one of the countless off-the-shelf bbs style boards? e.g. SimpleMachinesForum, phpBB etc Like my eevblog forum here:
This is how countless businesses and user group forums work. The more advanced pay-for ones can really be integrated into the look and feel of your corporate website. But so can the free ones too. Development support for these forums is excellent, and there are plenty of forum geeks you can hire to write custom scripts if needed. Zero spam, and you can appoint people to be admin/moderators if needed, so any bad stuff gets knocked on the head pretty quickly.
Almost every web host has these available as single click installs these days, otherwise it's pretty easy to donwload and install yourself. Trivial to set up and most users are familiar and comfortable with forums like these. Remember too, many users will not want to to use their real email address - ever.
Check out my Electronics Engineering Video Blog & Podcast:
I haven't found any that fit this set of requirements. :<
The OTS solutions (that I have seen) require *a* moderator (or are unmoderated) and require a "network visible" server. The solution I seek would *not* require a moderator and could be hosted anywhere mail is accessible (i.e., damn near everywhere) regardless of the size of the pipe.
Could you let any of these OTS solutions run AS IF an embedded system -- i.e. *unattended* yet *huge* uptimes?
You missed the criteria. *No* moderator. I.e., I am looking for a scheme whereby the users can moderate the forum (hence my reference to craig's list). So, any members misbehaving simply become un-members (i.e., strong incentive to behave as product updates are distributed via the same channel).
E.g., I belong to a few mailing lists (moderated) where membership is conditional on "good behavior". Getting booted is not something you want to have done as it costs you access to that resource (as well as your reputation! :> ).
OTOH, imagine USENET gated to your mailbox -- with all the cruft that comes with it :-/
Likewise, folks *in* this group probably won't want anonymous outsiders "eavesdropping" on their conversations. E.g., I am very careful, here, when I discuss problems to make sure I don't disclose anything that would betray a client's confidence (or get me fried for violating an NDA!). Since I can't know or control who reads my comments, I often have to "speak in parables", etc.
(i.e., I don't think Toshiba would want folks reading about "gas pedal questions" ~2 years ago... if you see the parallel)
You're presupposing you understand my application and user base better than I. :> Give me credit for having evaluated all the *obvious* solutions -- hence my post, here. If I'm looking for a five-wheeled vehicle, *assume* I *know* that I need a five-wheeled vehicle...
What I am looking for is a way to address the criteria that I have laid out; not to rehash why I have *decided* that those criteria define my solution space!
Their email addresses will never be exposed -- that's the whole point. If they don't wish to have any involvement, they don't have to participate.
OTOH, they have to "participate" even if they only want
*read* access ("membership" is given to customers only).
Sorry, I've never used Craigs List, but how can a user get booted without someone deciding they should be booted? You say NO moderator, but then you say users moderate the forum. Forums can have NO moderator if you so desire, or selected users can be moderators. Having a moderator on a BBS forum does not mean every post is moderated before it is posted, it just means moderators can chose to boot messages or people for misbehaving.
Membership of this forum is a condition of product updates? If so, that sounds like a bad move to me.
Yeah, that's how bbs forums work too. Any user who don't play by the rules either gets their messages cancelled, or booted altogether from the forum.
Yes, but you usually don't get that with BBS forums. Every one I belong to (and run) is civil and on-topic. No spam, no bad behaviour.
BBS forums can be completely private and "invite only".
Ok. But it is still completely unclear why you think a BBS forum will not work. BBS forums are pretty much the standard. People know them and understand how they work. You'd want a pretty good reason to use something else for a user type forum, and you haven't really explained why a BBS is not suitable.
What about those genuine customers that want to participate but don't ant to give you their real email address (even if it isn't released)? Users of my BBS forums can choose to hide their email addresss from everyone too, only I as the owner can see it.
Trivial to do with BBS forums, you can make them membership only.
Check out my Electronics Engineering Video Blog & Podcast:
Craigs List allows readers -- WHO CAN BE ANYONE -- to mark posts as inappropriate (for a variety of reasons). Some number of independant markings cause the post to be removed. (I've never explored the actual criteria they use).
The problem with such an approach is that the folks marking those posts remain anonymous. So, some small (?) group can prevent particular posts from appearing through concerted effort. And, their activities are never disclosed to others (all folks see -- *if* they are observant -- is that a posting
*was* there... and now its *gone*).
Imagine customer (company?) A deliberately conspiring to remove posts from customer (competitor!) B. No one would be the wiser! (Craigs List doesn't even require the reader's -- acting as moderators -- to disclose their identity to Craig's List).
Consider how that sort of approach would work (not) with USENET...
If you (I) don't want a moderator per se, then I am implicitly indicating that the members of the group must (somehow) fill this role. But, in such a way that they are "kept honest" by the other members of the group! (I can't imagine any sort of automated criteria that could fulfill this purpose... e.g., "if you boot 3 people, then *you* get booted automatically...)
I am suggesting *any* user can fulfill the role of moderator at any time -- or not. I.e., if the chatter rises to a level that annoys a particular user, then that user can step up and act to moderate that OT chatter. If that user abuses this capability (such abuse would be obvious to others in the group as they would be told of each such action), then the other users in the group can act on that "abusing" user.
[I am not suggesting this as *the* solution. I am merely trying to point to other "outside the box" ways of thinking of supplying the moderator functionality *without* a "Moderator"]
I am aware of that. See the Craig's List description above.
However, moderation is more important when using a push technology since the chatter ends up in your lap even if you don't go looking for it!
How does a customer know about updates if he doesn't want to be contacted?
But, the user can turn around and re-register as "John Dough" and no one is the wiser -- until he again abuses the forum. Ad infinitum. This contributes to the overall lowering of the quality/value of the forum.
OTOH, if you only get membership by "purchasing" the product, then you really only get one chance to screw up...
Shirley, you jest? I've been wandering around looking at various fora and see quite a lot of OT posts. Folks who just seem to "want someone to talk to". Would you call Tech Support "just to chat"? Would you call your plumber to chat about a problem you are having with your car?
Remember, I'm not talking about "highly technical" people. Look at how often The Left and Right rant, here, in what
*should* be a technical forum... consider how the chatter would (might?) change if you could boot individuals and they *couldn't* come back... For the most part, it's not a problem as you can simply ignore those posts (and persons).
Skilled users can build filters to remove the unwanted cruft. But, that puts unskilled users at a disadvantage -- they get stuck seeing all the cruft with little recourse.
Sure! As can mailing lists. And the membership can be hidden, public or revealed only to subscribers. The means for gaining "membership" can be open, invite only, or completely closed. None of this is new...
A BBS (assuming we aren't talking about accessing that via an SMTP gateway) is a pull technology. This means it has to accept requests whenever the user wants to access it. As such, it's hosting must be 24/7/365 (or, some other established set of "business hours").
Any server that has to be available "at will" is also exposed to hackery as it *is* accessible. Consider how much easier it is to protect an SMTP interface than an entire *host*! Traffic to the "server" servicing the "group" can be controlled by the server itself (i.e., *it* can decide when to go fetch its incoming mail and when it wants to *send* its outgoing mail). Imagine how much harder it is to hack a "Mail Station" than a "general purpose PC"... (the former has a very narrowly defined interface whereas the latter has to be adeptly managed to ensure it has the smallest visible interface possible)
And, it never knows how *many* users will want to access it at any particular time. As load increases, performance degrades. This degradation is directly visible to the user -- the interface becomes sluggish. (use web mail and see how performance varies throughout the course of the day!)
OTOH, using a "push" technology (like mail), allows the service to spread its workload out over time -- scheduling outbound mail and processing incoming mail as it sees fit. And, it can leverage a multitude of other hosts -- the mail hosts for each of its subscribers -- to act on its behalf (by contrast, a pull technology has to carry all of the load by itself).
Consider that the "server" can send one copy of a message to a multitude of recipients in a given MX. The "remote" MX can then handle distributing those copies to the N recipients that it services (instead of waiting for those N recipients to come *to* the "server").
Consider how heavily MS's servers get pounded whenever a new service pack is released -- everyone wants the update *now*. OTOH, if the updates were *pushed* to users, then MS could better manage the problem. Any effort "waiting" on a server is waste -- especially if you have to come back later to get what you want (and still have no assurances of being able to do so!)
A BBS requires state be kept for each subscriber -- i.e., which messages he/she has read, which ones he/she has replied to, which threads are of interest, etc. A mail client handles all of that overhead "magically".
Mail can be read via more interfaces than web pages can be browsed. E.g., you can read/send mail from a web based interface; you can't read web pages from a mail client! I've tried viewing several BBS's from a PDA and a smartphone and found both lacking (mail is "bad" on these devices but BBS's are abysmal as they have to fit within the context of a "web page"). I see this platform as a definite trend.
Providing "one to one" conversations via a BBS is clumsy. It's "something extra" that has to be implemented (maintained). OTOH, doing so with a push technology is trivial -- process the incoming message, rewrite the headers and pass it along to the intended recipient. There is no need to store it (state) on the BBS/server.
A mail client has an intuitive interface through which a user can preserve or ignore topics of interest (as well as tracking threads that he/she is participating in). Messages that are not of interest can simply be deleted. Messages that haven't (yet) been read, are notably indicated. etc. Doing the same in a BBS requires maintaining all of that state for each user *on* the BBS server (e.g., imagine if I want to read the first and fourth messages in a thread, delete/discard the third and leave the second unread).
There are very big differences in push and pull technologies. And their impacts on the service and its users. BBS's are The New News. But, they have all the same characteristics *of* News -- just a dressed up *remoted* interface.
Then, what do you define as "participating"? Should we USPS mail notices of updates to you? Are you willing to provide your name and STREET address? Are you willing to pay for a CD to be shipped to you with any software updates on it?
How are you going to ask questions? Call Tech Support? Or, should we provide you with a list of all of the telephone numbers of every other customer just so you can phone each of them to ask your questions? Should they be required to send transcripts of each question *they* pose in the forum to you via USPS mail?
You (the wanting-to-be-anonymous-subscriber) are, essentially, saying, "All of you folks should give up YOUR privacy just so I can keep *mine* (anonymity)"
Or, maybe you provide an option on your BBS: "Do you want your posts to be private?" If so, you end up with two separate fora -- one publicly visible and the other NOT publicly visible. And, you have to *hope* that the folks who have the answers to your questions are in the "visible" list! (wanna bet such a distinction would result in most of the "more knowledgeable" folks gravitating to the "more closed" list -- just so they didn't have to be bothered by the "simple" questions?)
And how is that different than what I am saying:
"Since it is too easy for folks to harvest email addresses, I think the software should hide email addresses completely. (I may also suggest creating arbitrary "handles" for users so that their real names are, by default, hidden -- unless they opt to disclose them deliberately)."
Are you sayiing that a user can opt to post as "Anonymous" at any time? Fine. Put the word "ANONYMOUS" in the first line of your (mail) message and the server removes your "arbitrary handle" so no one knows which "virtual person" you are. People can still reply directly to you -- without others seeing! -- by replying to *that* message. Having a "handle" is essential for tracking who said what (in a multiparty thread).
Sure, old technology. Modern BBS's are an outgrowth of much that originated in mailing lists.
They do especially if they have a web front end to it, which for the non computer savvy is the only interface they believe exists.
Also get put off as it is 'too technical', and want the web interface.
SMTP driven needs quite a bit of processing on each message to remove all sorts of problems. ...
Which considering the costs of webhosting these days is cheap and insignificant to set up, as even your own server with whatever on.
Anyway most folks who are not techies will still expect a web interface at least to do sign up.
You are joking aren't you?
An SMTP interface STILL means you have to close off other services and ensure that even DOS attacks, by flooding a closed http, telnet or other port does not slow service. It still has to receive packets to reject closed ports which takes up bandwidth restricting your valid service.
You could only get better protection by using a SMTP service on dial up to do a pull, to reduce visibility. Then again a ping attack with 64k blocks of data will slow you down.
Only if it is on Dial up, SMTP sits there waiting for messages just like HTTP servers sit there waiting for web requests.
I think you do not understand differences between sending email from a client using SMTP and the receiving end using SMTP, it has limited choice on receiving email. You would need somebody else's server to receive it and then fetch it from there like POP/IMAP works for email CLIENTS.
Do you realise how many scripts you are going to have to run to sanitise the emails, to remove unwanted types of headers like Reply Receipt and read receipts, let alone attachment, character set, virus/malware scanning and other isues.
What format are the emails to be as lots of emails get sent as plain text and/or HTML/RTF or even ascii encoded binary only.
Having of old run mailing list for technical users who could not drive their email software, and remote email servers around the world misconfigured, the work involved was not trivial.
Sorry the vast majority of 'mail stations' are general purpose PCs, maybe in a server hardware configuration, running some form of Nix, Windows or Windows Server, using the standard network stacks to talk to email server software like sendmail, postfix qmail, ftgate.........
So they have the same and MORE issues of security.
I think you are confusing SMTP as you see for sending email and how SMTP has to work on a server, i.e. be normally available
Standard network interface be it ethernet or dial up via PPP, that ends up as TCP/IP to the email server software.
You can generally ping mail stations.
That is true of both 'types' of system as they are basically the same.
Delays on SMTP throughput are possible when you get to thousands of users and making sure you are not seen as a Spamhost is sometimes difficult.
Response time to seeing messages posted appear, makes the service appear sluggish.
You really do not know what you are talking about.
You are talking about using somebody else potentially as a recognisable Spamhost. How to get your ISP to cut off your service.
Only those who blindly update, "if it aint broke don't fix it"
Real serious users actually wait for the masses to do MS's using the customer to test the release first.
The overhead of doing that by things like SMTP is ridiculous as the method does not support binary transmission, you have to ASCII encode your binary, bloating it and taking longer to send.
For weird ideas of 'magically'. BBS does not have to do all of those things if it does not want to, on a web interface it could just use the visited link colour changes to do that.
People moved away from discussion mailing lists as the general method of discussion about 6 years ago, as everybody went to web.
Lots of HTML and other emails are not displayable easily on PDAs either.
In my experience there are a lot of users who cannot effectively use their email clients, and you get 95% of users 'top posting' without trimming so as a thread progresses your bandwidth requirements increase exponentially.
They don't have to and mosts forums display nearly all messages in threads or only display the most recent threads or messages in a thread by date or quantity.
You have no idea of the facts of what will be required to maintain a SMTP type service, or the problems that can be involved.
That is what a lot of users expect even these days as a CD is easier to reload, when drives crash or systems are upgraded.
Most customers even today expect to contact tech support not other customers for a lot of products (washing machines to CAD software).
Most users of products rarely have the time or inclination to do that and will do so ONLY when they have a problem.
Their life (social or working) does not revolve around one product or its fora. Your working life may revolve around the product because that is your job, it isn't the customers.
Which you are trying to regenerate with an SMTP service but fail to see it.
Paul Carpenter | email@example.com
Don, I have been thinking along these lines myself recently. Just playing with the idea of adding a user forum, that is.
I must say I like the smtp idea. I guess my implementation would be a plain listserver with a command set - email driven - to suit your purposes. Voting can also be email driven. Now how this will work in todays spam age I am not sure, may be not at all, but then may be it won't be such a problem. Then the whole thing will be archived and instantly http accessible, perhaps the command set should also be http accessible. IOW newcomers will be able to read it like forums are read to get a feeling of where they are.
I guess much will be down to the actual command set and the anti-spam rules, the latter may have to be very hard for the thing to work (hence too annoynig for users... :-) ).
The thing that comes immediately to mind as far as the near-zero-effort-from-you moderation requirement is Slashcode.
NOTE: The Slashcode user-supplied moderation can become fanboy-driven.
"a list of providers who host slash sites, or design slash sites"
The SMTP requirement does put a huge kink into things. Simply stated: If you are going to have a site that accepts comments, you need some sort of CAPTCHA system in place. IMO, Slashdot and Wikipedia have very good ones (for not-signed-in folks).
So how would CAPTCHAs work with email (SMTP)? I suppose the forum could generate one and send it to a prospective poster, who would then have to send back the correct interpretation in order to post. Is that the sort of validation mechanism you had in mind?
You were wrong, and I'm man enough to admit it.
- a Usenet "apology"
They are just two that I run, but it's the same thing on almost every other forum that I frequent. People are always civil enough to post in the appropriate group category.
Oh, for goodness sake, people like to talk on forums! Sometimes it gets off topic etc, what's the big deal?, that's the nature of the beast.
Yeah, that's how countless forums operate. What's the problem with that? If it goes down for a few minutes or an hour every now and again, no one really cares. I'm not talking about SMTP access, I'm talking about a BBS forum people visit.
Really? I run half a dozen high bandwidth sites with databases, streaming video, and two forums on a simple OTOH, using a "push" technology (like mail), allows the
Gawd, I'm not even going to read all that you wrote there, it's getting too much.
But I really get the impression you are MASSIVELY over analysing this whole thing, really.
Good luck getting your ideal solution. If it was me I'd just stick up a private BBS forum and be done with it.
Check out my Electronics Engineering Video Blog & Podcast:
I think you have some very fundamental misconceptions about servers, security, SMTP, hacking, hosting, etc., - the whole package, really.
Setting up a server that is accessible on the Internet via some service (http, smtp, whatever) is a risk if you don't know what you are doing, but simple to do securely if you /do/ know.
Clearly, you are in the "don't know" category. That's not meant as an insult - no one can be an expert at everything. The trick is to know what you know, and know what you don't know - and get outside help when you need something you don't know about.
It's possible that you can learn enough by sitting down and discussing things with a knowledgeable person (it's much easier to do that than to discuss it in a newsgroup). But you are probably better off looking for an external hosting service. Then you don't have to worry about security - it's an SEP. All you need to concentrate on is the features for the users.
If you are really insistent on using SMTP, the solution has been in use for decades - it's called a "mailing list". Personally, I prefer mailing lists (or newsgroups) to web forums - but I don't know if I am typical of your target audience.
Interesting task. There are two well done webforum & NNTP=20 combination systems that i have seen: OpenSuse forums and=20 Microsoft public forums. Both are actively managed. Yahoo=20 forums also appear to have SMTP gateways, but they overdid=20 the registration thing. Registration is a good idea anyway,=20 it could require reading an email and using a key (good for=20 a year?) in order to post (anti-bot). Regular spam blocking=20 will also have to be considered, as well as automatic=20 blocking of profanity. Take a look at STUMP robomoderation=20 program and group distributed moderation. For people that=20 wish to continue to private email they should have the wit to=20 take anti-harvesting precautions (don't block it, but don't=20 encourage it). Provide a FAQ and post some policy basics=20 regularly. The FAQ should contain some "how to make a good=20 post" information.
With the criteria stated it may be that having free open source=20 software modified to suit is the way to go on this. Get=20 something with a BSD or similar license if you can. A GPL=20 license may bite you.
As far as I can see, it wouldn't. "Fools ignore complexity. Pragmatists suffer it. Experts avoid it. Geniuses remove it." --Alan Perlis's Programming Proverb #58
I see the email part of this as limiting, clumsy, and unnecessary. The suggestions in this thread of using a pre-baked forum solution remove that.
The suggestion of Slashcode in my response also addresses the issue of moderation. Anyone familiar with a site running Slashcode knows about thresholds for viewing. How well that moderation works does involve achieving a certain critical mass of users.
Jones and Brown have also indicated that there is a fundamental misunderstanding of the issues. The one big thing that I see missing (aside from my mentions of Slashcode and CAPTCHA) is a specific means of avoiding spammers
A straight forward listserver is easy to do. Start with something like MajorDomo and tweek it.
The problems lie in the "other features" that I described. Implementation is easy -- the problem is figuring out a good model that users will feel comfortable with and that fits their "KISS" expectations. Folks don't want to read instructions...
You can always put a filter on the front end to keep cruft out of the "list". And, close the list so only subscribers have access to it. I certainly wouldn't want to discuss trade secrets in such a forum... :> But, for "user driven support", I think it makes sense (i.e. don't say anything you wouldn't want someone to read!)
Having control over readership (subscribership) does a lot.
The *best* interface would be NNTP. But, I don't think that is a mechanism that is as ubiquitous as email.
I think keeping spam out can be accomplished by limiting the "senders" who can access the list. And, if the membership and their email addresses are hidden (sanitized from messages), then even someone with access to the "archives" (*you* want them to be public; that isn't consistent with my goals) couldn't determine the right email address to spoof to gain entry!
The problem then boils down to keeping subscribers from misbehaving. Hence the idea of letting them police themselves -- and, police their own policing! :>
I've been working on a smart "news filter" that could play some role, here. But, it takes a while to train; it works well in high volume applications as there are lots of data points for it to operate. But, a product forum tends to see more sporadic activity -- fewer posts from a greater number of "members" -- who are constantly changing. Hard to train something in that sort of environment :-/
I've separate thread starting that hopefully will show how something like this can be done... and, how it can be scaled.
Hmmm... I'll have to see if I can steal any ideas there. I think the problem is that "product support" forums tend to have more transient membership. I.e., folks post *a* problem. Wait for *the* answer (naively hoping each problem *will* have "an answer"). Then disappear.
Some folks inevitably hang around and become the "gurus".
And, others hang around just to harrass folks ("Why do you want to do *that*? You should do this instead...")
Obviously, you want the gurus to hang around -- they are a resource that costs you (the vendor) nothing! They get some satisfaction out of helping. Or, perhaps, enjoy the recognition that it affords them.
Conversely, you want to *quickly* get rid of the bullies, ranters, etc. They degrade the quality of the support and the overall experience. They also have a negative effect on the gurus -- as they often compete for attention with those folks (hoping to drive them away, etc.).
Unfortunately, getting this type of sporadic user to
*rate* the quality of their experience (i.e., the messages they are receiving back from the forum) is hard. There is no incentive for them to do so -- unless it is very easy/intuitive.
E.g., if you (meaning the forum software) could decide how the user *acted* on the information presented ("Yes, this was helpful to me", "No, this guy missed the boat entirely", "Jeez, this guy is a real *sshole!", etc.) then you could adjust the way the subscribers are treated (e.g., bounced).
Yes. Otherwise, I would already have a solution and wouldn't need to ask! :>
I think SMTP and control over the subscriber list (email addresses) deals with the access problem. No need to worry if this is a robot or a human -- as long as the subscribers don't complain about "its" posts! :>
You don't *accept* them in the first place! It's not an "open forum". You can't just "sign up". If you put in place a mechanism by which they can be *booted* (or, at least *silenced*), then they get one chance to screw up (how that one chance is defined -- is it one post? twelve posts? ten people who you have annoyed?) before they are silenced.
(No, you can't just "resubscribe" -- that's the whole point!)