Subject: missing values

From: Jakson Aquino <[email-masked]>
Date: Sun, 6 Feb 2005 18:46:14 -0300 (ART)

Hello All!

I didn't look at the source code carefuly, but from
the "behavior" of Statist it seems to me that we have
a problem. 

Suppose that a research team applied a questionnaire
to 600 people, asking them 50 questions. Some question
wouldn't apply to every one, and many people wouldn't
answer many questions. Now, I have a database with 50
columns and 600 rows, and many missing values. Every
time I want to do some correlation between variables I
will have to do:

1) Open with Statist a version of the database with no
'M' inside. The missing values would be indicated by
some valid value.

2) Save the columns that I want to correlate in a
ASCII file.

3) Quit Statist

4) Use an external program to recode the new and
smaller database to turn into 'M' all values that, in
fact, are missing.

5) Load the new small recoded database with the option
-delrow and, then, run the analysis. 

While still becoming familiar with the database, I
should run many analyzes, mainly regression analyzes
or multiple linear correlations, with different
combinations of variables. It's easy to imagine how
boring this work would be.

Statist always deletes missing values while reading a
database. The option -delrow mean only that the entire
row, instead of just one data point, must be deleted.
That's the wrong approach because the rows can be
broken without the option -delrow, and valid values
can be deleted with the option -delrow. Data in the
same row are supposed to be a single case, and they
can't be split in different rows. Thus, rows with
missing values should not be deleted. They should be
ignored during calculations, when necessary, not while
loading a data file. And the problem can be more than
a mere inconvenient; it can be a bug if we don't
follow the five steps described above, and, unluckily,
have one of the two circumstances described below:

  1) A data base is opened without the option -delrow,
the columns used in a calculation have missing values,
and, for coincidence, they have the same number of
missing values, but not in the same rows. The bug
happens because Statist only tests whether columns
have the same number of data points before proceeding.
In this case, the correlation for these columns would
proceed and Statist would report some wrong result.

  2) The option -delrow is used, but the analysis
don't uses all columns, and the columns don't have
their missing values at exactly the same rows,
consequently some rows that should be used in the
analysis were deleted while opening the data base.
Again, Statist would proceed with the analysis and
report some result.

We need some way of keeping Statist aware of missing
values, instead of simply deleting them and testing
whether the columns have the same number of data
points. Perhaps we can define the biggest possible
double as missing value! [I don't know what the
biggest double value is.] That is, whenever Statist
finds 'M' in the database it puts 9.99999e+99 in its
memory, and whenever it finds 9.99999e+99 in its
memory it assumes that it is a missing value. While
writing columns as ASCII files, missing would come
back to the usual 'M'. And, just for precaution, if it
finds 9.99999e+99 while reading an ASCII file, it will
reports a error. Of course this will slow down the
computations, since an if(xx[i] == 9.99999e+99) will
be necessary when reading each single data point from
memory. And, the worst, it would be necessary to fix
all functions! Any better solution?

Anyway, we need to write a documentation in English
and include this information in the documentation.

Am I wrong, and there is no problem with the way
Statist deal with missing
values?

Best regards,

Jakson

__________________________________________________
Converse com seus amigos em tempo real com o Yahoo! Messenger 
http://br.download.yahoo.com/messenger/

---

From: Bernhard Reiter <[email-masked]>
Date: Mon, 7 Feb 2005 11:08:34 +0100

Hi Jakson,

> the "behavior" of Statist it seems to me that we have

I agree that the behaviour you describe below is problematic
and we need to test it more thoroughly and find a solution.

> -delrow and, then, run the analysis. 

My idea was that recoding the long file
with 'M' on the missing values for the full database
should be a one step operation and then statist can 
do all the various analysis on it.
If this is not the case, this is indeed bad.

Could you produce a few example files so that
the error can be made obvious. We can use that as test cases
when we fix the bug.

> points. 

Yes.
This could be done before the calculations.

> Perhaps we can define the biggest possible
> biggest double value is.] 

There will be a standard define for this.

> That is, whenever Statist

Putting a special value in is not the niced solution,
however I would need to look at the code to think about
others.

> and include this information in the documentation.

An English documentation is really needed, that is true.

	Bernhard

---

From: Jakson Aquino <[email-masked]>
Date: Mon, 7 Feb 2005 20:29:19 -0300 (ART)

>The ideal solution would be that someone makes a
special version 
>of gettext so that we can extract the German
translation directly
>from the source. ;)

Gettext has an option to extract messages from ( , ).
I used it to build the de_DE.po and es_ES.po. And I
learned it from the statist Makefile, where we have:

xgettext --keyword="_:1"

I guessed that -k_:1,2 would extract both English and
German, and it did work! German became msgid_plural. I
had just to replace msgid_plural by msgstr.

>You can put in the list or me as interims translator,
>because of course I do speak German. ;)

I'll have to update the potfiles because I found about
10 messages that were only in German (I used something
like grep out_r\(\" to find them). They don't need
translation from German to English, but need in other
languages. I also made a rough translation of a few
commentaries that still were in German in the sources.
Perhaps we'll have more people reading and commenting
the source code if it is entirely in English. 

I don't know Statist code very well yet, but it seemed
to me that it would be no problem if Statist saved the
columns in ASCII files with their names. So, a third
thing that I did was to add 4 lines to menue.c. Now,
when I open a file containing columns saved by my
version of Statist the columns are already with their
names. I'll submit a new patch with these changes
together with the revised .po files, including (I
hope) the Italian one.

>It sometimes takes a while until I have time to visit
changes.
>At Intevation we want to set up a new platform for
projects like
>this based on Gforge, then contributions will even be
easier.

Sometimes I also take a few days to answer emails
because I'm without Internet in my house. I'm using
either cybercafes or the computers of my uni.

>Makefiles have build in rules for c files,
>but this denies the dependencies, so it better shall
be in.

Statist only compiled under DOS after I added the rule
to memory_handling.o. My idea was to show Statist to
some friends under DOS and tell them that they could
see the program running in Portuguese under Linux.
None of my friends has ever used Linux.

I'll prepare a small data base to make it easy to see
the problem with missing values. The first thing that
I did after compiling Statist for the first time was
to run a multiple linear correlation with 4 variables.
I used a database that comes with SPSS, and run a
linear regression in SPSS with the same 4 variables.
Each column had originally 1500 rows, but they dropped
to about 925 after -delrow did its work. In SPSS, the
"N" was also 925. The results (coefficients, etc) were
exactly the same, but I had to follow the steps
described in my previous email to run the analysis in
Statist.

I also opened, but did no test, with a database that
has 17 columns and 350000 rows. The for variables were
extracted from PNAD (Brazilian National Household
Survey).

Best regards,

Jakson

_______________________________________________________ 
Yahoo! Acesso Grátis - Instale o discador do Yahoo! agora. http://br.acesso.yahoo.com/ - Internet rápida e grátis

---

From: Jakson Aquino <[email-masked]>
Date: Tue, 8 Feb 2005 16:46:58 -0300 (ART)

Hi Bernhard!

>the error can be made obvious. We can use that as
test cases

I'm  sending the example files as attachment. The file
results_gss explains what was done and presents the
results. The file gss20x6_file_info has information
about the very small database used to make the test.

>however I would need to look at the code to think
about

Yes, if the special value was present in the database
as a valid value this would be a problem. Perhaps a
option like --diffsysmis=number (define different
system missing value, other than 9.99999e+99) could
solve the problem. PSPP, which is cloning spss, seems
to use a list of user defined missing values to each
colum:

[jakson@localhost src]$ grep MISSING *.h -n
exprP.h:187:    OP_STR_MIS,                     /*
MISSING(strvar). */
var.h:282:    MISSING_NONE,             /* No
user-missing values. */
var.h:283:    MISSING_1,                        /* One
user-missing value. */
var.h:284:    MISSING_2,                        /* Two
user-missing values. */
var.h:285:    MISSING_3,                        /*
Three user-missing values. */
var.h:286:    MISSING_RANGE,            /* [a,b]. */
var.h:287:    MISSING_LOW,              /* (-inf,a].
*/
var.h:288:    MISSING_HIGH,             /* (a,+inf].
*/
var.h:289:    MISSING_RANGE_1,          /* [a,b], c.
*/
var.h:290:    MISSING_LOW_1,            /* (-inf,a],
b. */
var.h:291:    MISSING_HIGH_1,           /* (a,+inf),
b. */
var.h:292:    MISSING_COUNT

PSPP also seems to have a special value reserved to
sysmis. Look at the following lines:

var.h, line 40 and ss.:
/* Special values. */
#define SYSMIS (-DBL_MAX)
#define LOWEST second_lowest_value
#define HIGHEST DBL_MAX

magic.h, line 29 and ss.:
fndef SECOND_LOWEST_VALUE
/* "Second lowest" value for a flt64; that is,
(-FLT64_MAX) + epsilon. */
double second_lowest_value;
#endif

PSPP code is too big and complex to me, and I just
used grep to find the word "missing". But it seems
that they are using the biggest possible double
negative number as sysmis, and a lot of other user
defined missing values. Perhaps using a very big (and
negative) number was not so dangerous. Anyway, I don't
know if this is a good solution.

Anyway, I don't think it is necessary to complicate
the code creating a complex list of user defined
missing values. I prefer the simpler approach of
Statist of simply putting a 'M' in the database. In
social research we have three main sources of abudant
missing values: (1) people don't know how to answer a
question; (2) they do know how to, but are unwilling
to answer, and (3) the question don't apply. In some
analysis, all three cases are better considered
missing values, but in others, often using the same
data base, "I don't know" and "I prefer don't answer
this question" must be distinctly counted and
analysed. But this would happens only rarely, and it
would not be that difficult to write a program to
automatically recode an original database and create a
new one with the 'M's correctly replacing values that
have to be recoded as missing. 

Best,

Jakson

---

From: Andreas Beyer <[email-masked]>
Date: Wed, 09 Feb 2005 09:39:31 +0100

Hi,

instead of using something like -99 as a representation for missing 
values we should use something like infinity. The GNU C-lib knows the 
constant:

float NAN;

which is "not-a-number". Unfortunately this constant is only defined on 
GNU systems. In addition, NAN is unequal to itself. Hence, the following 
is true:

float x = NAN;
assert(x != x);

Instead we might use the maximum double, i.e. something like the following:

#include float.h // DBL_MAX

static double _no_value = DBL_MAX;

/* Obtaining the internal representation for a missing value. */
double missing_value()
{
    return _no_value;
}

/* Check if val is a missing value. */
int is_missing_value(double val)
{
    return val == _no_value;
}

I prefer encapsulating the missing-value check in a function, because 
that leaves room for future extensions that might become more complex.
	Andreas

Jakson Aquino schrieb:
>>the error can be made obvious. We can use that as
> 
> test cases
> 
>>when we fix the bug.
> 
> 
>>however I would need to look at the code to think
> 
> about
> 
>>others.
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Statist-list mailing list
> [email-masked]
> https://intevation.de/mailman/listinfo/statist-list

---

From: Jakson Aquino <[email-masked]>
Date: Wed, 9 Feb 2005 09:29:21 -0300 (ART)

Hi!

Before reading the email written by Andreas Beyer, I
had written something similar. I also wrote some
suggestions of code, but I don't know Statist source
code sufficiently well to have any certainty that what
I wrote would work.

If I'm not wrong, we can fix the problem of missing 
values writing few lines of code. I'm still thinking 
that we can define one specific double number as
system missing value.

I - DEFINING SYSMIS

First we would have to give to Statist the ability to 
keep missing values. We could borrow one line from
pspp and put in statist.h:

I wrote the following program to see the exact value 
of -DBL_MAX:

#include <stdio.h>
#include <values.h>

int main(){ 
  printf("\n%e\n\n", SYSMIS);
}

The program was compiled wit g++, and the output
was: -1.797693e+308

The complete number is:

-17976931348623157081452742373170435679807056752584499
659891747680315726078002853876058955863276687817154045
895351438246423432132688946418276846754670353751698604
991057655128207624549009038932894407586850845513394230
458323690322294816580855933212334827479782620414472316
8738177180919299881250404026184124858368.000000

Certainly the probability of finding this specific 
number in a database is very small. We can tell in 
the documentation that if this number appears in the 
database it is the responsibility of the user to 
define a different SYSMIS with the option 
-sysmis=number, or something similar.

II - WRITING SYSMIS IN TEMPFILES

Now that we have the variable SYSMIS, we have to read
columns correctly. Perhaps the following replacement 
will be enough:

data.c, line 261 and ff:

if ((token[0] == NODATA) && (strlen(token)==1)) {
  colread ++;
}
else if (sscanf(token, "%lf", &test)==1) {
  FWRITE(&test, sizeof(REAL), 1, ttempfile[actcol]);
  nn[actcol] ++;
  colread ++;
}

Should be changed to something like:

if ((token[0] == NODATA) && (strlen(token)==1)) {
  FWRITE(&SYSMIS, sizeof(REAL), 1, ttempfile[actcol]);
  nn[actcol] ++;
  colread ++;
}
else if (sscanf(token, "%lf", &test)==1) {
  FWRITE(&test, sizeof(REAL), 1, ttempfile[actcol]);
  nn[actcol] ++;
  colread ++;
}

III - READING SYSMIS FROM TEMPFILES

At first, I thought that it would be necessary to 
rewrite all functions because of the missing values 
problem. But now I have some hope that this will not 
be necessary. If I'm guessing correctly, a code 
similar to the following can solve the problem:

After the following code (data.c : 482 and ff.):

  for (k=0; k<n_alloc; k++) {
    if (!x_read[acol[k]]) {
      xx[acol[k]] = readcol(acol[k]);
      label_tab[acol[k]].ptr = xx[acol[k]];
      label_tab[acol[k]].str = alias[acol[k]];
    }

We can add something similar to the following 
to delete rows with missing values:

  int cr = 0; /* current row */
  int tr = 0; /* total number of rows already checked
*/
  BOOL RowHasMis = FALSE;
  while(tr < total_number_of_rows_in_data_base){
    for (k=0; k<n_alloc; k++)
      if(xx[acol[k][tr] == SYSMIS) RowHasMis = TRUE;
           
    if (RowHasMis){
        tr++;
        RowHasMis = FALSE;
    }
    else{ 
      for (k=0; k<n_alloc; k++)
        xx[acol[k]][cr] = xx[acol[k]][tr];
      cr++;
      tr++;
    }
  }
  current_valid_number_of_rows_in_xx = cr;

All code above is only a rough idea. I wrote it only 
here, and didn't test nothing in the real source code.
Probably Statist source code already have variables 
that correspond to
"current_valid_number_of_rows_in_xx" 
and "total_number_of_rows_in_data_base".

Of course Andreas Beyer's idea of putting the code in
a function is better than adding it directly to this
function. I think that when this fuction is written
the option -delrow will no longer be necessary.

*****************************************

It would be very nice if the people in this mail-list,
including those that no longer use Statist, told us 
about the difficulties they have had using Statist. 
This would help us to know what to do before starting 
writing code.

*****************************************

The main difference with Andreas Beyer suggestion was
that I'm now following pspp solution and proposing a
negative number as missing value.

Who will try to fix the problem? Andreas Beyer?

Best,

Jakson

---

From: Jakson Aquino <[email-masked]>
Date: Thu, 10 Feb 2005 18:13:35 -0300 (ART)

Hello All!

Did anyone worked on the problem of missing values?

I was curious to know whether the code that I wrote in
the other email would work, and, then, I tested it.
With some adaptations, it seems that it's working.
However, I tested it only with <Miscellaneous | Mean,
standard deviations etc.> and <Multiple linear
correlation>. I opened the file that I sent as example
(gss20x6_M.dat), and Statist correctly deleted rows
with missing values during many calculations. Latter,
I also tested new code with the database that have 17
columns and 350000 rows. It worked well also.

I declared REAL SYSMIS as (-DBL_MAX + 1) because
Statist is already using the values (-DBL_MAX) and
(DBL_MAX).
 
The changes that I made are below. I can send them as
a patch against version 1.0.1, but perhaps it's better
to wait for the solution of the  problem with gnuplot.

=================================================
statist.h
-------------------------------------------------

extern REAL SYSMIS; /* 'M's are put in tempfiles with
this value */
extern int     tn;  /* total number of rows in data
base */

=================================================

=================================================
statist.c
-------------------------------------------------

REAL SYSMIS = (-DBL_MAX) + 1;
int tn;

=================================================

=================================================
data.c
-------------------------------------------------
in function readsourcefile(), lines 262 and ff.

	 if ((token[0] == NODATA) && (strlen(token)==1)) {
           FWRITE(&SYSMIS, sizeof(REAL), 1,
ttempfile[actcol]);
           nn[actcol] ++;
	   colread ++;
	 }
	 else if (sscanf(token, "%lf", &test)==1) {
	   FWRITE(&test, sizeof(REAL), 1, ttempfile[actcol]);
	   nn[actcol] ++;
	   colread ++;
	 }
       . . . 

       tn = lread - first_line;

-------------------------------------------------
void alloc_cols(int n_alloc) {
  int k;
  for (k=0; k<n_alloc; k++)
    nn[acol[k]] = tn; /* tn = total number of rows in
database */

  /* deleting all columns */
  for (k=0; k<MCOL; k++){
    if((x_read[k])){
      myfree(xx[k]);
      x_read[k] = FALSE;
      label_tab[k].ptr = NULL;       
    }
  }

  /* putting selected columns in memory */
    if (!x_read[acol[k]]){

  /* deleting rows with missing values */
  int cr = 0; /* current row */
  int tr = 0; /* total number of rows already checked
*/
  int RowHasMis = 0;
  while(tr < tn){
    for (k=0; k<n_alloc; k++)
      if(xx[acol[k]][tr] == SYSMIS) RowHasMis = 1;
           
    if (RowHasMis){
        tr++;
        RowHasMis = 0;
    }
    else{ 
    nn[acol[k]] = cr;
    
  if (log_set)
    for (k=0; k<n_alloc; k++)
      fprintf(logfile, _("Variable %i = Column %s\n",
            "Variable %i = Spalte %s\n"), (k+1),
alias[acol[k]] );
}
=================================================

* * * * * * * * * * 

As you can see, the function alloc_cols() has two work
arounds with the variable nn; one at the begin of the
function, and the other at the end.

Best,

Jakson