Sorry, Wombat, I've never heard of "Split-Testing" before & obviously cannot directly comment on it!!

However, if it was just a simple choice of presenting some colours to one set of potential customers (with a different set of colours to others) then I could understand the purpose of it!

But, (REGARDLESS of what others might see) if I have to jump through obtuse "hoops" just to find basic information then it is very unlikely that I would become a customer!!

That, IMHO, is simple "common-sense" & I wouldn't have thought (whatever fancy name it might be called) that there was any need to "test" this!

Nothing against split testing, but for those who want to look up product details having no access without resorting to Google and site:www.plus.net is pretty poor.

Split testing is often colours or just changing the emphasis and order on a page.

Actually I just had to do that on the BT Retail website to check all the details when re-doing our listings with their latest changes and roll-out of BT Sport packages.

Still not good.

In reply to a post by MrSaffron:

Split testing is often colours or just changing the emphasis and order on a page.

Mr S, as you mention in another post - nothing against "Split-Testing" if it really is about variations in layout and/or colours.

But where it is to make certain Users jump through hoops, then I really don't see the point!

In reply to a post by RobertoS:

Well it may be common, but I've never encountered it before.

The truth is that you just haven't been aware of it before.

It happens online, in print, on TV and radio. Even in direct mail. Anywhere where marketing people can put two different "messages" in front of parts of their audience, and test the result.

Some tests are focussed on a specific product, and whether you choose to buy or not. This could include offering you different ways to pay.

Some tests are focussed on a variety of choices presented to you... to see which choice you pick out of the group presented (and to see if varying the group of offers makes any difference).

Some tests are just focussed on the overall impact to traffic (eg vistors to websites, or footfall in physical shops).

1) Is it also common for one or more versions of the website to completely omit any path to certain products? Specifically those with a lower overall profit margin? Not even a clue to their existence? Understandable with trial flavours of KitKat, but broadband?

Marketers know that offering choice can be a problem: confused customers don't buy.

So offering a choice can be good for some customers, but bad for others. Which is best for the company overall? Should a choice be removed? Or offered in a totally different way?

Therefore tests to show the impact of completely removing a choice are perfectly valid propositions.

2) If I went to a site a few times to check certain aspects against a competitor I was considering, and found not only a different view sometimes but information I knew I had seen shortly before was no longer there, I would run a mile from such apparent incompetence.

Valid point. Offline adverts do this by varying (for example) in different TV regions, or on different radio stations.

Online testing requires the website to take care that you see a consistent behaviour across multiple visits. Cookies are a likely part of the solution.

Using different browsers, or different computers, makes it easy to see different variants - and some people will see this. But most won't (as you haven't, before today).

How difficult has it been to overcome a historic bad reputation, and how easily will this tactic destroy all that hard work? The tactic, nay - the strategy, needs immediate binning before it has the wrong effect.

That was my initial reaction, as an engineer, to seeing what marketers did. Now I understand it, and why it is done (selling is, in large part, psychological).

With understanding, I can temper my reaction to any one encounter, based on whether I think a company is doing something fundamentally wrong (such as omitting something in the advert (ala the ASA ruling).

In this case, what are PN actually doing wrong? They are failing to offer one of their product variants, in an easy-to-find way, to some of their website visitors. But they haven't made it unavailable to anyone, or discriminated in any way. Just made it hard to find.

Ultimately, the tests will determine what is the right way to offer the variants... whether that is on the home page, a product page, or the sign-up journey. Then PN will stick to the best way, and move on to test something else.

Isn't that better than having a marketer guess at which way works best, and sticking with that method come hell or high water?

Of course, there is a thin line here. If the tests show a departure from credibility, then more people will walk away because of a lack of trust, rather than just failing to like the offer enough to buy it.

I don't think the PN test here is enough to cause mistrust.

In reply to a post by 4M2:

In reply to a post by bpullen:

Has been since the back-end of last year. Monthly contracts typically cost an extra £2.50pcm.

Would it be possible to include that in the "legal bit"? I had to find that out from here http://www.plus.net/support/broadband/products/low_c... !!!

I'll ask about and see if we can get it added.

In reply to a post by johnjburness:

Bob, Not certain how I would take a screen-grab & then post it here!!

However, just typing in Plusnet to Google takes me to:-
https://www.google.co.uk/search?q=plusnet&ie=utf-8&o...

To which I note the following:-

Ads related to plusnet

plus.net - Plusnet Official Site‎
www.plus.net/Plusnet
1/2 Price Unlimited Broadband for 9 Months - Limited Time!‎
1/2 Price Broadband
1/2 Price Fibre Broadband

Plusnet Business Broadband
Award-Winning Broadband

As you can see, there is nothing that links the "1/2 Price" to a requirement to have the Phone Service included!

Thanks, I'll run that by the legal folk. Cheers for pointing it out.

In reply to a post by RobertoS:

1) Is it also common for one or more versions of the website to completely omit any path to certain products? Specifically those with a lower overall profit margin? Not even a clue to their existence? Understandable with trial flavours of KitKat, but broadband?

It's not that radical a thing to do as part of a split test.

In reply to a post by RobertoS:

2) If I went to a site a few times to check certain aspects against a competitor I was considering, and found not only a different view sometimes but information I knew I had seen shortly before was no longer there, I would run a mile from such apparent incompetence.

Which again, is something we've taken into consideration.

In reply to a post by RobertoS:

How difficult has it been to overcome a historic bad reputation, and how easily will this tactic destroy all that hard work? The tactic, nay - the strategy, needs immediate binning before it has the wrong effect.

I don't personally think our split testing (which we've done before remember) will significantly tarnish our reputation. I could be wrong of course and we appreciate everyone's feedback. As I think I mentioned in an earlier reply, those involved in the intricacies of the split test have their eyes on this thread.

Rgds,

In reply to a post by WWWombat:

what are PN actually doing wrong?

Plusnet place an Advert promoting 1/2 Price Broadband (WITHOUT any reference to the need to sign up for Phone Service as well), which then links to an offer that combines the 2 Services. In fact the very fact that you can obtain Broadband only is carefully hidden &, when "discovered", you still cannot find any reference to 1/2 Price Broadband.

Personally I would regard that as very questionable Advertising & (whether intentional or otherwise) very misleading!

In reply to a post by WWWombat:

I don't think the PN test here is enough to cause mistrust.

IMHO, "questionable Advertising" will always cause mistrust!

In reply to a post by johnjburness:

But where it is to make certain Users jump through hoops, then I really don't see the point!

I can't say what Plusnet's reasoning was, but there are some ways I could explain it...

One possibility is that someone in PN wants to know what the signup rate will be if they ONLY display the options to take broadband with the phone line. Perhaps they are thinking of making this the only way to sign up. Or perhaps they just want to establish a base number, before performing further tests with differing degrees of visibility of the "broadband without phone line" option.

Intuitively, you'd say that more people will walk away (because, like you, they'd have to hunt for the details they want, or just don't hunt at all). On the other hand, people who were going to move phone line anyway wouldn't be affected at all.

In the middle, however, is a group of people who weren't originally intending to change phone line, and wouldn't do so if they were presented with the "broadband only" prices first. But, with only "broadband with phone" prices in front of them, they perhaps reconsider, and sign up... bringing more income each month.

Result: PN would lose some potential customers, but gain some others. Or perhaps the ones they do gain bring slightly more income with them. Does the improved income from one group offset the reduced income from the other group? Which is best?

You say the results are intuitive. That PN should follow common-sense. But that is only "common-sense" from your personal perspective. Everyone else has a different perspective, and different intuition applies to them and their behaviour.

Split-testing works by testing the statistical behaviour of many people.

The fact is that split-testing, done properly, does not always agree with intuition or common-sense.

For me split testing can be a pain as I tend to visit a site on what ever device is closest to hand which may be iOS, android, osx or IE9, chrome, firefox.

I know I am not the demographic, but then ignoring my role on tbb I am the sort who tend to make recommendations when neighbours ask about who is best at the moment etc