Hi Richard,
Well, I would agree with you for web logs....whoever thought that you could make money out of analyzing clicks on pages seemed to miss the whole opportunity of the media....I know some companies did 'cowpathing'
etc and redesigned sites to avoid drop offs etc.....but the whole thing completely missed the opportunity available to companies via the web...
If I may indulge the group....
Some colleagues and I have been hard at work 'inventing the future' of customer dialogs, one touch point being web pages......and our little band of brothers is something to behold.....our fearless leader is Sean Kelly who’s books and white papers are listed here:
We see the dialog with the customer as THE most important aspect of managing customer relationships over time and THE most likely tool for modifying customer behaviour.......'modify customer behaviour' was a concept a group of us came up with about 10 years ago....
What we have developed now looks like 'magic' along the Arthur C. Clarke lines of thinking....."Any technology sufficiently advanced is indistinguishable from 'magic' ".
We are now able to allow business users to develop and design customer dialogs and feedback loops/offers....these dialogs can be imbedded into any web based application as the dialog is merely an iframe that can be put onto a page.........no IT people required.... ;-)
We have also built 'a segmentation engine for the rest of us' which mere mortals can use......not like some others even I struggle with!!
Business users can use the segmentation engine to segment their clients.
The dialogs the customer engages in places them into segments over time and the feedback/offers they receive are customized according the customers interactions over time and other information fed into the segmentation engine.....we are even doing segmentation on-line in real time and sending back customized longitudinally aware feedback during the interaction as a popup/popunder....we can also send feedback via any mechanism that can be talked to by a computer...which is pretty much everything...
So...where's the business value...??
This kind of very advanced use of web based interactions enables a company to interact with the customer like the 'corner store' of the past.....the company really does get to know the customer and their wants and needs again following 30 years of disintermediation via the use of technology...the profit results of our early sites have been, quite literally, 'off the charts'.....We are still working on it and we are still perfecting it...but we are all sure of one thing.....we have embedded all that we can think of for the management of customer dialogs and segmentation for profit improvement and we are all absolutely convinced that this thing will be something of a 'license to print money' through the ability to develop and maintain customer relationships through sensitive dialogs proposing what is of value to the client when the client needs what it is the company has to offer.....
So....there is a very, very large amount of money to be made by companies effectively using their web sites and this web data belongs in the EDW as one of the most valuable piece of information of all.... :-)
Just my thoughts!!!
Best Regards
Peter
-----Original Message-----
From: richard.bailey via dw-select
[
Sent: 04 August 2006 18:00
To: pete...
Subject: RE:[dw-select] Re: AW: Re: RE: RE: DTS - is it viable?
Web data don't get me started!!! oh its to late.
Web data unless you’re an ISP should go no where near your data
warehouse
1) It is inaccurate there are three way to track users through
web sites cookies, data tagging and IP address
Lets start with cookies these are rubbish estimates suggest that
there is only a 80% to 90% penetration rate. That’s certainly my
experience with it anyway.
Data tagging very good but it will only tell you that a link was
clicked not that the page returned, there is also a debt ongoing
about about the size of website that it can successfully tag.
IP tracking for normal people forget it, unless your an ISP.
2) The data is the worst data you can possibly feed into a data
warehouse platform. why? Well, if uses can type stuff in then
your problems are just starting, developers tend to hand code.
The log files don't format correctly and once the portal team
gets access to the data your problems just get worse.
3) Data value: - let’s look at the data that you can actually get
from a log file
IP address
Browser type
Page requested
Page from
Status code
Oh and if you have some cookie data.
As you would undoubtedly agree really useful data!! you then need
to apply ABCe standards
Traffic!! Seems like a lot of effort for not a lot of valuable
data. checkout
Unless you suffer like me, we process 12 million hits a week and
I am made to process that into the data warehouse but at least I
can enrich it as an ISP we know where you are!!!
---------------Original Message---------------
>Martyn,
>
>Assume you decide for full click tracking DW on a website,
including
>checkmark and radio button recording. The website has 5000
pages, tracking
>is by userid or cookieid.
>It will become a huge task to change all the pages over to click
tracking,
>while the ETL, DW part are fairly simple.
>Kalman Toth, Database, Data Warehouse & BI Architect
>URL:
in the World
>----- Original Message -----
>From: "martyn.richard.jones via dw-select"
<dw-select@Groups.ITtoolbox.com>
>To: "SQLUSA" <technicalsqlusa@earthlink.net>
>Sent: Friday, August 04, 2006 3:36 AM
>Subject: AW: [dw-select] AW: Re: RE: RE: DTS - is it viable?
>>
>> Hi,
>> If the main issue is getting data into the DW and not the
actual
>> performance of querying the data then I guess one must be
unfortunate
>> enough to:
>> a. Want to load a vast amount of data
>> b. in a short period of time
>> c. on inadequate platforms
>> Only for all that "valuable data" to be used by a well reduced
business
>> population.
>> As the person responsible for architecting the first TB data
>> in Europe (1996) you can rest assured, that at least here, we
don´t
>> have decades of experience with Relational DBMS products that
supported
>> any serious partitioning options; and even in late 96 we were
just
>> talking about view partitions and not entire object
partitioning.
>> As I alluded to earlier, if you have a 6 hour window to load
all
>> required data into the DW and it takes you 3 (start of window
+ 3) hours
>> with product A and 4 (start of window + 4) hours with product
B, what is
>> the problem for ths business?
>> BTW I don´t think Grace Hopper included materialized views in
her
>> definition of ANSI COBOL.
>> Hope that helps, have fun,
>> Martyn Richard Jones
>> Data Warehouse Architect
>> -----Ursprüngliche Nachricht-----
>> Von: SQLUSA via dw-select
>> Gesendet: Donnerstag, 3. August 2006 18:14
>> An: Jones, Martyn Richard
>> Betreff: RE:[dw-select] AW: Re: RE: RE: DTS - is it viable?
>> >I think these days a lot of the efficiency / performance
requirements
>> >are still in getting data out of the DW, the querying of data
in the
>> DW,
>> >and questions of partitioning, striping, object dirtribution,
indexing,
>> >statistics for cost/rule based query optimization,
materialized views
>> >etc
>> >
>> Actually I disagree. The above is covered by several
well-known
>> technologie
>> s and we have decades of experience since the time COBOL.
>> I see the main issue as getting the data into the data
warehouse.
>> ETL-ing i
>> s only one part it. The other part is changing current
applications so
>> they
>> can actually "emit" data.
>> Kalman Toth
>> -----Original Message-----
>> >From: brett via dw-select <dw-select@Groups.ITtoolbox.com>
>> >Sent: Aug 3, 2006 10:37 AM
>> >To: SQLUSA <technicalsqlusa@earthlink.net>
>> >Subject: RE:[dw-select] AW: Re: RE: RE: DTS - is it viable?
>> >Awesome reply....
>> >---------------Original Message---------------
>> >>Hello,
>> >>
>> >>I think there is a set of fundamental issues that need to be
addressed
>> >>in terms of Data Warehousing and ETL.
>> >>As I see it, the reasons for using an ETL in the first place
is not
>> for
>> >>maximum performance of the process, so a tool is chosen that
will can
>> be
>> >>used to:
>> >>a. develop the required extraction, transformation and
loading
>> processes
>> >>in the expected / required time-frame.
>> >>b. to reliably and timely execute the ETL, which accurately
loads data
>> >>into the DW / Data Marts in the production environments,
within the
>> >>required processing windows
>> >>c. generate most of the ETL code automatically
>> >>d. easier to maintain, support of metadata, etc.
>> >>The thing is, if your load process window is, for example
from
>> midnight
>> >>to 7:00 AM then if you can execute everything by 4:00 AM or
3:00 AM is
>> >>neither here nor there, increasing the load process
efficiency won´t
>> >>win you new customers, and having something that takes ten
minutes
>> >>rather than five minutes to run in the ETL part of the DW
world, wont
>> >>lose you customers either.
>> >>I think these days a lot of the efficiency / performance
>> >>are still in getting data out of the DW, the querying of
data in the
>> >>and questions of partitioning, striping, object
dirtribution,
>> indexing,
>> >>statistics for cost/rule based query optimization,
>> >>etc
>> >>Even then, as most Data Warehousing should be focused on
strategic
>> >>decision support, even if your senior execs can´t use the DW
in
>> days,
>> >>it shouldn´t really have any serious impact on the business.
For
>> data
>> >>warehouses that are used to provide data to compliment or
even support
>> >>CRM and Core systems, then one must take a different
approach, but
>> then
>> >>again, what absolutely current mission critical data would
one wish to
>> >>keep in the DW rather than also in the core business
applications?
>> >>At the end of the day, if you can produce performance
improvements of
>> >>100% there are questions that still needs to be asked, such
as:
>> >>a. So what?
>> >>b. How much will it cost the business to implement?
>> >>c What secondary impact and cost will the new tool have on
the
>> business?
>> >>d How long will it take?
>> >>e What are the risks, what are the likelihood of those
risks
>> occurring,
>> >>how often, and how much?
>> >>d. Just what is the bottom line business benefits if we
decide to
>> >>switch?
>> >>e. Just what are the bottom line business disadvantges if we
>> >>Everthing must be accompanied by figures: USD, dates,
probabilities,
>> >>etc.
>> >>Hope this helps,
>> >>Martyn Richard Jones
>> >>Data Warehouse Architect