Subject: locale.h and gnuplot

From: Jakson Aquino <[email-masked]>
Date: Thu, 10 Feb 2005 09:37:47 -0300 (ART)

Hello All!

When trying to see a boxplot I got the following
gnuplot error:

Spalte fuer Variable 1: TVHOURS

Erstelle gnuplot-Graphik ...

0 vermutliche Ausreisser gefunden

   --- Bitte mit ENTER fortfahren ... ---
gnuplot> plot [0,8:5,2][9:13]
'/home/jakson/stat_gpl.dat' with lines
                ^
         line 0: ':' or keyword 'to' expected

The problem is that, in many languages (among them:
German, French, Spanish, Italian, and Portuguese), the
decimal separator is not a "."; it's a ",". However,
the culprit is <locale.h>. With locale.h, the printf
function becomes very smart. It's good to see the
results of calculations printed with the correct
decimal separator, but, while writing a file to
gnuplot, locale.h is a problem.

Another --far less important-- side effect of locale.h
is that if we save columns in an ascii file running
Statist in a language other than English, it will not
be possible to open the file running Statist in
English. We would have, for example, 1,30000e+01
instead of 1.30000e+01. Since this side effect isn't
properly a bug of Statist, I think that it will be
enough to document it. And the solution is very
simple: use some tool to replace all "," with ".".

Is there any easy way of fixing the problem with
gnuplot?

Best,

Jakson

_______________________________________________________ 
Yahoo! Acesso Grátis - Instale o discador do Yahoo! agora. http://br.acesso.yahoo.com/ - Internet rápida e grátis

---

From: Bernhard Reiter <[email-masked]>
Date: Thu, 10 Feb 2005 13:45:24 +0100

> Is there any easy way of fixing the problem with

We probably need to ignore 
the locale settings for files and gnuplot.
This is possible, I only have to look up how.

---

From: Andreas Beyer <[email-masked]>
Date: Thu, 10 Feb 2005 15:55:53 +0100

I think I had the same problem with StatistX. If I remember correctly, 
my solution was to set the locale to the default C-locale before passing 
anything to gnuplot. Afterwards I set it back to the previous setting.

Andreas

Bernhard Reiter schrieb:
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Statist-list mailing list
> [email-masked]
> https://intevation.de/mailman/listinfo/statist-list

-- 
--------------------------------------
   Dr. Andreas Beyer - [email-masked]
   http://www.imb-jena.de/~beyer
--------------------------------------

---

From: Jakson Aquino <[email-masked]>
Date: Thu, 10 Feb 2005 18:15:24 -0300 (ART)

Hi,

I'll look at StatistX source code.

Best,

Jakson

---

From: Jakson Aquino <[email-masked]>
Date: Fri, 18 Feb 2005 18:46:59 -0300 (ART)

Hello All!

StatistX macro solved the problem with gnuplot. I
always called the macros inside the condition #ifndef
NO_GETTEXT. I deleted the following lines from the
macro, but only because they seems to be written in
Objective C and I don't know how to translate them.

if (SX_saved_locale == NULL) { \
 [NSException raise: NSMallocException \
   format: @"set locale: Not enough memory."]; \
} \

I'm sending the patch against the cvs version of
Statist with all changes that I made since February 4.
I'm not a professional programmer, and I'm not the
owner of Statist, thus, although my version of Statist
is running properly, you might find it better to make
some changes to improve the code or to make it more
consistent with Statist "philosophy".

The initial idea of simply deleting rows with missing
values before calculations would be good enough for
social scientists, but other people might have
different problems. Now there are three ways of
calling the function alloc_cols() (and, thus,
getcols(), which calls alloc_cols()):

   - TRUE: delete rows with missing values.
   - FALSE: delete missing values, but not the
         entire row if we have more than one row.
   - 3: let the column as it is. That is, let 
         it with its missing values.

I used the following rule to decide whether to use
getcols(i, TRUE, FALSE or 3):

  - If in the original source code getcols was 
    followed by "if ((!empty) && (equal_rows(2)))",
    I used TRUE.
  - I also used TRUE if only one column was being 
    read from tempfile (In this case the result is 
    the same as FALSE).
  - If getcols was followed only by "if (!empty)",
    I used FALSE
  - for <DATA MANAGEMENT | List data of one column> 
    and almost the entire menu DATA MANIPULATION, 
    I used "3" (don't delete missing values). 

In order to "Read another file" with column labels, I
turned the variable n_alias into global, and this
created a bug. Every time Statist was automatically
creating a new column (like the "resi" after a
regression, or out_ after deleting outliers etc...)
the number of columns (ncol) was updated, but not the
number of column labels (n_alias). Then, if we tried
to read a new file without deleting the old data, bug!
However, I soved the problem adding a lot of n_alias++
to many functions.

Another problem, already present in the cvs version of
statist, happened when I tried to open a file with 4
columns (two with 10 rows and two with 12 rows). The
file was created by Statist with the option "save
columns in ASCII file". But Statist couldn't
understand " .       " as a valid number. I fixed this
 problem. Now, Statist writes 'M's to complete the
smaller rows.

I ran practically all Statist menu entries, and
everything seems to be OK. The only segmentation fault
that I got was due to an error in my pt_BR.po.
Sometimes I also ran Statist simultaneously in
Portuguese and English, to compare results and
graphics.

One doubt that I have is whether defining missing
value as (-DBL_MAX) can cause problems in functions
that already use the value REAL_MIN, defined in
statist.h, and that also is (-DBL_MAX). My attempt of
defining missing values as (-DBL_MAX + 1) didn't
really changed the value of -DBL_MAX.

I wrote a draft of an future English documentation.
It's incomplete, and the English is not very good, but
at least someone who can read English will understand
it. I used a translator to get an idea of what Dirk
Melcher wrote in the German documentation.

I have a lot of work for the next days. Thus, this is
my last contribution for the next days (or weeks).

If anyone has any doubt about any of the changes I
made, please tell me. Some of the changes were
explained in previous emails.

Best regards,

Jakson

> https://intevation.de/mailman/listinfo/statist-list

__________________________________________________
Converse com seus amigos em tempo real com o Yahoo! Messenger 
http://br.download.yahoo.com/messenger/

---

From: Andreas Beyer <[email-masked]>
Date: Mon, 21 Feb 2005 09:40:40 +0100

>    format: @"set locale: Not enough memory."]; \
When you store the current locale information, the program could 
potentially run out of memory. You have to check that, which StatistX 
does by testing the condition
   SX_saved_locale == NULL
If this condition is true, the program has no memory left (or something 
else went wrong). statist should perform the same test at the same place 
and handle the problem in some way. E.g. the following is a possible way 
to deal with it:

if (saved_locale == NULL) {
    /* Run out of memory. Fatal, stop the program. */
    out_err(FAT, ERR_FILE, ERR_LINE,
          _("set locale: Not enough memory."));
}

Look at "statist.h" for details about out_err(). If possible, one could 
also use mymalloc() from memory_handling.h.

	Andreas

---

From: Bernhard Reiter <[email-masked]>
Date: Mon, 21 Feb 2005 22:50:04 +0100

> anything to gnuplot. Afterwards I set it back to the previous setting.

Another way to solve this is to only set the LC_CTYPE and LC_MESSAGES
locales and keep all printf and scanf functions to use the "C" locale instead.
(Because this is a small change, I just checked it into CVS.)

The question is: What do we want?

Gnuplot command: needs the "C" locale for sure.

Data files: I would hope to keep them interoperable internationally,
so I tend to prefer "C" locale here, too.

Number display during interactive use: When the datafile is international
I also think it is reasonable to keep the numbers printed during the
interactive sessions international.

There is another point in that people that are used to the old statist
might be surprised by the new behaviour if we also changed LC_NUMERC.

Jakson, Andreas: What do you think?

	Bernhard

---

From: Andreas Beyer <[email-masked]>
Date: Tue, 22 Feb 2005 09:14:32 +0100

I guess the American standard (== C-locale) is virtually an 
international standard. So, we might go for the C-locale in case of 
LC_NUMERIC. In any case this should only be the default! The makefiles 
should have an option to set the locale explicitely. Thus, in order to 
communicate with gnuplot we would still have to set the C-locale 
explicitely (and set it back afterwards). The makefile should allow for 
something like

# set this to 0 to use your local number representation in data files
USE_C_LOCALE 1
[...]
gcc [...] $(USE_C_LOCALE)

There could be people relying on the fact that statist uses their own 
locale when reading data. E.g. MSExcel uses the country specific 
settings. If someone exports data from there, he/she will not 
necessarily get files that are compatible with the C-locale.
Anyway it seems better not to change the entire locale when sending data 
to gnuplot. Otherwise also (error-)messages would be printed in English 
during this phase. That would be a petty after all these translation 
efforts. ;-) Thus, we should definitely only change LC_NUMERIC.

	Andreas

---

From: Bernhard Reiter <[email-masked]>
Date: Tue, 22 Feb 2005 10:19:22 +0100

Am 22. Feb 2005 um 09:14:32 schrieb Andreas Beyer:
> LC_NUMERIC. In any case this should only be the default! 

This is the current situation in CVS, 
after I read up on that locale documentation.

> The makefiles 
> should have an option to set the locale explicitely. 

Even better we could make it a command line option, 
then people can decide each time they call statist.
Or do you think that would be worse?

> Thus, in order to 
> explicitely (and set it back afterwards). 

Yes and LC_NUMERIC should be enough for this.
Anybody wants to do the patch? :)

	Bernhard

---

From: Jakson Aquino <[email-masked]>
Date: Tue, 22 Feb 2005 11:31:13 -0300 (ART)

Hello All,

I tested the new cvs version of Statist, and setting
LC_TYPE and LC_MESSAGES, the accents are being
correctly displayed. But I still prefer to see the
decimal separator being a comma when the output is in
Portuguese. My suggestion is to set the locale to "C"
before reading data files and writing columns, and
re-set it back to saved_locale after these operations.
Only plot.c needs a lot of set and reset, but I
already did it in the patch of last week. 

If this solution is accepted, I think that we have
only to set & reset locale in two other places:
readsourcefile() (data.c) and menue.c ("Data
management | Export columns as ASCII-data"). 

The Statist documentation should warn the user that
the data file uses "dots" as decimal separators, but
that the results are internationalized.

We could also add a menu option like "Data management
| Export columns as ASCII-data using current locale".
The documentation would explain what this means.

The last week patch regarding the problem with missing
values was made against the old cvs version of
Statist. I don't have experience with diff, but since
the cvs was modified, and because you made corrections
in some of my changes, I guess that the patch is no
more applicable. If this is the case, I can prepare
the patch again. That is, I will

 - run diff to compare the current cvs version 
   with my version of statist.
 - manually do in my version of statist the 
   corrections that you did in cvs.
 - run diff again

Jakson

Best,

Jakson

 --- Bernhard Reiter <[email-masked]>

> ATTACHMENT part 1.2 application/pgp-signature

---

From: Jakson Aquino <[email-masked]>
Date: Tue, 22 Feb 2005 11:33:57 -0300 (ART)

Hi Andreas,

Thank you for completing the code. Next time I send a
patch, I can make the changes. I think that the code
below now is correct, right?

#ifndef NO_GETTEXT
char * SX_saved_locale, * SX_old_locale;
#define SX_SET_LOCALE	\
	SX_old_locale = setlocale (LC_NUMERIC, NULL); \
        SX_saved_locale =
mymalloc(strlen(SX_old_locale)+1); \
	strcpy(SX_saved_locale, SX_old_locale); \
	setlocale  (LC_NUMERIC, "C"); 

#define SX_RESET_LOCALE \
	setlocale (LC_NUMERIC, SX_saved_locale); \
	free(SX_saved_locale); SX_saved_locale = NULL;
#endif

I think that we can put the above lines in statist.h.

Best regards,

Jakson

---

From: Bernhard Reiter <[email-masked]>
Date: Tue, 22 Feb 2005 17:32:05 +0100

Am 22. Feb 2005 um 11:31:13 schrieb Jakson Aquino:
> Portuguese. 

What do you think of a command line option?

> that the results are internationalized.

Yes, in any case: It needs to be documented.

> The documentation would explain what this means.

Good idea.

> more applicable. 

It should be doable for me to apply the patches.
Patch uses a fuzzy algorithm and still works when other parts 
of the file have been changed.

	Bernhard

---

From: Jakson Aquino <[email-masked]>
Date: Tue, 22 Feb 2005 16:56:02 -0300 (ART)

Hi Bernhard,

> What do you think of a command line option?

The user that I imagine is some student of social
science that is migrating from Windows. He will use
Statist only some times in an year, and will have
problems memorizing commands. I think that it's a good
idea to have the options in command line, but I also
think that the same options can also be in some menu
option like "Main menu | Options". The options could
be toogled on/off from the menu.

My preference is that, as its default behavior,
Statist saves database in "C" locale, and shows
results and saves logfiles with LC_NUMERIC set to
current locale. The user would interact with a program
fully running in its language, but databases in "C"
locale has some adavantages: portability among people
in different countries, and compatibility with
gnuplot. People who knows gnuplot commands can save
columns as ASCII file and do something with them using
gnuplot without any need of changing the standard
behavior of Statist.

> of the file have been changed.

Very good! It would be boring to repeat the same work
again :)

Best,

Jakson