Subject: [Thuban-list] visual classifier

From: "Moritz Lennert" <[email-masked]>
Date: Thu, 4 Dec 2003 10:22:07 +0100 (CET)

On July 11, 2003, Daniel Calvelo wrote:
>In the attachment you will find a set of files and a patch against what was
>the June 20th CVS. They provide the beginnings of a custom classifier.
Let me
>elaborate.
>
>The patch adds a "Custom breaks" option to the class generator. When
>selected,
>it opens a graphical display of the selected field that includes: a
histogram
>(a slider controls the number of bins used), a "pointgram" (a point, or
>rather
>a vertical bar for each data value) and a set of triangular markers
>indicating
>breaks. These markers can be deleted (double-click) and added
(single-click >on
> new location). They represent cut points from which a set of ranges will be
>generated.

Has this been taken any further ?

Moritz

---

From: Jan-Oliver Wagner <[email-masked]>
Date: Thu, 4 Dec 2003 10:26:41 +0100

> Has this been taken any further ?

it looked promising but was a bit buggy. Don't know whether Daniel
worked further on it.

-- 
Jan-Oliver Wagner               http://intevation.de/~jan/

Intevation GmbH	              	     http://intevation.de/
FreeGIS	                               http://freegis.org/

---

From: "Daniel Calvelo" <[email-masked]>
Date: Thu, 4 Dec 2003 08:41:44 -0500

Hi all.

Sorry, I haven't had time to work on the thing. 

Furthermore, the current infrastructure should be overhauled to accomodate
this kind of "extension", and the whole will have to be rewritten when
transparency is considered.

If there is interest, I could update the patch to work with recent CVS, and
maybe add the most urgent features. (Yes, I'm very happy somebody's actually
interested!)

Cheers,

Daniel.

-- Daniel Calvelo Aros
-- Dirección General de Información Agraria
-- Ministerio de Agricultura del Perú
-- (51-1) 424-9001

---------- Original Message -----------
From: Jan-Oliver Wagner <[email-masked]>
To: Thuban Mailing List <[email-masked]>
Sent: Thu, 4 Dec 2003 10:26:41 +0100
Subject: Re: [Thuban-list] visual classifier

> _______________________________________________
> Thuban-list mailing list
> [email-masked]
> https://intevation.de/mailman/listinfo/thuban-list
------- End of Original Message -------

---

From: Jan-Oliver Wagner <[email-masked]>
Date: Thu, 4 Dec 2003 14:49:02 +0100

Hi Daniel,

> maybe add the most urgent features. (Yes, I'm very happy somebody's actually

if it is not too much work for you (or you just love this work :-)
it would be nice to have an update for current CVS. If possible,
an extension would be nice, but I am not 100% sure right now
whether this is achievable. An extension could go immediately
to CVS (marked as experimental).

Best

	Jan

---

From: "Moritz Lennert" <[email-masked]>
Date: Thu, 4 Dec 2003 17:37:23 +0100 (CET)

> maybe add the most urgent features. (Yes, I'm very happy somebody's

I am very interested. In the aim of getting my department to slowly move
towards free software solutions, a cartography solution is one of the main
issues. And data classification is an important element in that. It would
actually be ideal to have an automatic "natural breaks" classification
available in addition to the current uniform distribution and quantiles.
When you said:

> and a set of triangular markers indicating
> (single-click on new location). They represent cut points from which a set
> of ranges will be generated.

does that mean that these markers are set automatically at the beginning ?
How do you define these breaks ?

Moritz

---

From: "Daniel Calvelo" <[email-masked]>
Date: Thu, 4 Dec 2003 12:47:00 -0500

["Moritz Lennert"]
> does that mean that these markers are set automatically at the 
> beginning ? How do you define these breaks ?

Yes, they are set initially as evenly spaced. I looked hard at how you could
initialize them using the other classifiers, but the infrastructure ties very
closely the classifiers with the class definitions, and I couldn't find an
elegant, unobtrusive solution.

My idea was to have a permanent View of the classifier (in the cute world, it
would also have visual cues for the corresponding drawing style), in which
markers could be gridded, reset to automatic settings (quantiles, even breaks,
"natural" breaks,...), hand-tuned either visually or by input, and so on. A
kind of classification studio if you wish. Generating the theme classes would
be either a side-effect or an extra action for this subsystem.

I got stuck by wxWindows too. I couldn't produce something correct that gave
me sliders that I could drag and drop. Could wx-savvy people give any hint on
that?

I also explored natural breaks. I found very scarce literature on the subject.
>From what I gathered, natural breaks are based on a kernel approximation to
the data distribution. From previous experience, programming these things
needs tuning and lots of special casing. If you have ideas on the subject,
they are most welcome.

Daniel.

---

From: "Moritz Lennert" <[email-masked]>
Date: Thu, 4 Dec 2003 19:41:34 +0100 (CET)

> needs tuning and lots of special casing. If you have ideas on the subject,

We have a fortran program here which was written ages ago and which uses a
geometrical approach, exploring the slope between data points. If you are
interested I can send it to you (I will have to check with the authors,
but I would be very surprised if they weren't willing to release the code
freely).

Moritz

---

From: "Daniel Calvelo" <[email-masked]>
Date: Thu, 4 Dec 2003 13:59:18 -0500

If you could get the code, I will try to include the algorithm. It might be
better to first implement it as another classifier and then work on the visual
part.

Do you reckon any licence issues?

Daniel.

-- Daniel Calvelo Aros

---------- Original Message -----------
From: "Moritz Lennert" <[email-masked]>
To: "Daniel Calvelo" <[email-masked]>
Cc: [email-masked]
Sent: Thu, 4 Dec 2003 19:41:34 +0100 (CET)
Subject: Re: [Thuban-list] visual classifier

> We have a fortran program here which was written ages ago and which 
> uses a geometrical approach, exploring the slope between data 
> points. If you are interested I can send it to you (I will have to 
> check with the authors, but I would be very surprised if they 
> weren't willing to release the code freely).
> 
> Moritz
------- End of Original Message -------

---

From: Frank Koormann <[email-masked]>
Date: Fri, 5 Dec 2003 09:50:42 +0100

> but I would be very surprised if they weren't willing to release the code

Just for completeness or if some problems arise from using the fortran
code: The original paper on natural breaks ("Jenks
optimization") is "Error on Choroplethic Maps: Definitions,
Measurement, Reduction", G. F. Jenks & F. C. Caspall, Annals of the
Association of American Geographers 61 (2), pp. 217-244 (1971)

Should be available via JSTOR (http://www.jstor.org/journals/00045608.html)
if somebody has an account there.

Best regards,

        Frank

-- 
Frank Koormann                             <[email-masked]>
 Professional Service around Free Software       (http://intevation.net/)
 FreeGIS Project                                 (http://freegis.org/)

---

From: "Moritz Lennert" <[email-masked]>
Date: Fri, 5 Dec 2003 10:00:27 +0100 (CET)

Frank Koormann said:
>> >>From what I gathered, natural breaks are based on a kernel
>> approximation
>> >> to
> Should be available via JSTOR
> (http://www.jstor.org/journals/00045608.html)

I do and I downloaded the article if anyone is interested.

Moritz

---

From: "Daniel Calvelo" <[email-masked]>
Date: Fri, 5 Dec 2003 20:25:08 -0500

I am interested!

BTW I looked into the fortran code you sent me. AFAICT, it's based on
recursive partitionning of the dataset according to a measure of the
goodness-of-fit with straight lines. I'm translating it into Python to test
further; I'll keep the list informed.

I'll try it also with K-means: a long time ago I did that myself. Thanks for
the feedback!!!

Daniel.

-- Daniel Calvelo Aros

------- End of Original Message -------

---

From: "Dan Putler" <[email-masked]>
Date: Thu, 04 Dec 2003 14:38:51 -0800

Hi Daniel,

The "natural breaks" method as done in ArcView is also known as Jenks Method.  One citation that might help in your literature search (given the references it contains) is Jenks, George F. and Fred C. Caspall (1971), "Error in Choroplethic Maps: Definition, Measurement, Reduction," Annals of the Association of American Geographers 61 (2, June), 217-244.

Jenks methods is actually based on a method proposed by Walter Fisher.  The cite to Fisher's work is Fisher, Walter D. (1958), "On Grouping for Maximum Homogeneity," American Statistical Assoication Journal, December.  Fisher's method turns out to be pretty computationally intensive, so my guess is that ESRI actually doesn't use it in ArcView.  Ultimately, "natural breaks" amounts to single variable cluster analysis.  Given this, a K-Means cluster analysis algorithm could be used.  Another approach (that is used as an alternative approach to Fisher's method in xlstat, a set of add-in multivariate statistics macros of Excel) is an algorithm proposed by Anderberg (Anderberg M.R. (1973). Cluster analysis for applications. Academic Press, New York).  A benefit to using K-Means to create the "natural breaks" is that the SciPy package already contains a K-Means method function written in Python.  Although, it may require bringing in a lot of bagage to Thuban.

Some thoughts.  Given the level of detail, I've been thinking about this.  I wrote two classifiers for OpenEV, and wanted to include a natural breaks classifier.  However, I haven't had time to work on it, and don't think I will for awhile.

Dan

*********** REPLY SEPARATOR  ***********

>needs tuning and lots of special casing. If you have ideas on the subject,

_________________________________________
Dan Putler
Sauder School of Business
The University of British Columbia
Email: putler at sauder dot ubc dot ca
Phone: 604-822-8329

---

From: Jan-Oliver Wagner <[email-masked]>
Date: Sat, 6 Dec 2003 11:28:05 +0100

On Thu, Dec 04, 2003 at 02:38:51PM -0800, Dan Putler wrote:
> A benefit to using K-Means to create the "natural breaks" is that the SciPy package already contains a K-Means method function written in Python.  Although, it may require bringing in a lot of bagage to Thuban.

Thuban will treat such bagage in the same it handles the other ones:
If the packages are found, the functionality is available, else
disabled.

So actually not a too serious problem :-)

Best

	Jan