Subject: recode, get sample, and get columns

From: "Jakson A. Aquino" <[email-masked]>
Date: Fri, 24 Jun 2005 07:51:24 -0300

Hello All!

I developed some tools for Statist while trying to
use the PNAD database. PNAD is a national survey
made by the Brazilian government almost every
year. The resulting fixed-width database has more
than 300 variables and 360000 cases, and it
probably is the most popular database among
Brazilian social scientists.

I created a program to extract variables from a
fixed width database (xcols.c), another to extract
a sample of the rows of a Statist data file
(xsample.c), and a third one to recode a Statist
data file (recode.c). I also wrote the first draft
documentation for the three programs. Although
many researchers in the natural sciences might
never need any program like these three, I believe
that they are undispensable for a social
scientist.

If the new code from recode.c, xsample.c and
xcols.c is added to Statist, we can give to
Statist users two options: (1) invoke Statist with
options -r, -s, or -x, and (2) use the menu to
choose "recode", "create a sample" or "extract
columns". xcols.c and xsample.c are very small,
and if they are added to Statist, they can become
functions of data.c. recode.c is bigger and more
complex, and, probably, it would be better to keep
it as a separate file.

If the code is accepted as part of Statist, at
least some small adaptations are necessary, like
using out_r, mymalloc, FREAD, and FWRITE, instead
of printf, malloc, fread and fwrite.

Comments and bug reports are welcome!

Best,

Jakson

---

From: Bernhard Reiter <[email-masked]>
Date: Thu, 11 Aug 2005 16:47:01 +0200

Hi Jakson,

Am 24. Jun 2005 um 07:51:24 schrieb Jakson A. Aquino:
> use the PNAD database. 

first: Thanks again for these nice efforts.

> one to recode a Statist
> documentation for the three programs. 

> recode.c is bigger and more

I had a peek at the documentation and have two comments:

	As you can see from --bernhard, I tried to recreate new long
        options with the convention that they should start with a "--"
        and not just a "-". For the new options we might as well use "--"
	as their is no need for backward compability.

        I ask myself if using "awk" or python for this kind of reformatting
        would be better as it is more general and not much more verbose.
        Or the other way around: What is the advantage of using Statist
        for this?

Best,
	Bernhard

---

From: Jakson Aquino <[email-masked]>
Date: Fri, 12 Aug 2005 08:31:44 -0300

Hi Bernhard,

Em Qui 11 Ago 2005 11:47, Bernhard Reiter escreveu:
> 	As you can see from --bernhard, I tried to recreate
> new long options with the convention that they
> should start with a "--" and not just a "-". For the
> new options we might as well use "--" as their is no
> need for backward compability.

I agree! Since the first time I saw the statist help 
text that I perceived the existence of two styles of 
options. I don't know why I chosen the old style for 
the new options. Perhaps because the majority of 
options are in the old style. What a bad criterion!

What about if we turn all options to the new style of 
using "--" for full option name and "-" for 
abbreviations? The old style could still be accepted by 
produce a warning that it is deprecated and will no 
longer be supported in future releases. Only the new 
style would be documented.
 
>         I ask myself if using "awk" or python for
> this kind of reformatting would be better as it is
> more general and not much more verbose. Or the other
> way around: What is the advantage of using Statist
> for this?

You are correct! I simply don't know how to program 
with python, and I didn't know the existence of awk 
until some days ago. That's why I put some tips on the 
usage of awk and other gnu core utils. I discovered 
awk while trying to compile statist under Windows. 
Reading GnuWin32 download page I had interest in 
downloading core utils, and reading core utils 
documentation I found "awk" and "join". I even thought 
about removing recode.c from statist, but I think that 
the syntax of a "recode_config_file" is still simpler 
than the syntax of awk, at least for people who don't 
know how to program in C. Of course the development of 
recode.c wouldn't have been a priority if I already 
knew awk.

I still didn't find a tool for merging databases. I 
tried "join", but the results weren't satisfactory.
Perhaps I don't have enough knowledge on how to use 
join.

Best,

Jakson

		
_______________________________________________________ 
Yahoo! Acesso Grátis - Internet rápida e grátis. 
Instale o discador agora! http://br.acesso.yahoo.com/

---

From: Bernhard Reiter <[email-masked]>
Date: Fri, 12 Aug 2005 15:30:32 +0200

Hi Jakson,

> style would be documented.

Changes in statist have been slow for a long while
and there is no immedeate reasons to force all scripts to
the new options format.
I would say that we should in any case use the long format
for the new long options.
We could add "--" versions for the old long options, too
and have the warning that at some unknown point in the future
the "-" versions will be deprecated.

> until some days ago. 

I did not now that you did not know. 
There is a book called
	Brian W. Kernighan & Rob Pike, 
	The Unix Programming Environment, 1984, Prentice-Hall.

which teaches a lot about how to use small commands to assemble a
more complicated operation.

> I even thought about removing recode.c from statist, 
> but I think that the syntax of a "recode_config_file" is still simpler 
> know how to program in C. 

The question to consider is: Does the additional effort of learning
statist's syntax and maintaining the code help more then
learing recepies in a standard language?
If yes, you can keep the code. 
Otherwise removing it, will keep statist cleaner.

Usually standard languages have knowledge that is useful in other
circumstances as well. This is why I prefer python in many cases now,
though smaller operations tend to be longer than in awk.

> Perhaps I don't have enough knowledge on how to use 

Can you describe what you want to do?
As far as I can see "join" will need a matching keyword in both tables.
Maybe you want "paste"?

     $ cat num2
     1
     2
     $ cat let3
     a
     b
     c
     $ paste num2 let3
     1       a
     2       b
             c

Best,
	Bernhard

---

From: Jakson Aquino <[email-masked]>
Date: Fri, 12 Aug 2005 19:52:59 -0300

Hi Bernhard,

> the "-" versions will be deprecated.

It seems that we have only to add the options inside the optrec[MOPT] (in 
statist.c). For "help", for example, we have:

{"-h", &help}, {"-help", &help}, {"-?", &help}

In this case, it's just a matter of adding one more "help": {"--help", &help}, 
and doing the same with the other variables. And, then, update the value of 
MOPT. We have only to choose the abbreviation letters. Suggestions:

- -s --silent
- -l --log
- -p --noplot
- -f --nofile
- -t --thist
- -B --bernhard
- -x --xsample
- -c --xcols
- -b --nobell

> 	The Unix Programming Environment, 1984, Prentice-Hall.

Thanks for the reference. On Monday I'll look for it or for something similar 
(if it's not there) in the library.

> though smaller operations tend to be longer than in awk.

Your reasons are strong. I only regret that I lost my time developing recode.c 
and you lost yours translating its verbose messages. I agree with your two 
arguments: awk is useful for tasks other than manipulating statist data 
files, and recode.c is potentially a source of bugs. One of them you already 
found. I'm going to made the necessary changes in the source, and in the 
manual. Regarding Python, I would have to learn it. I'll try it latter, when 
I have the need of a big script. For now, I'm very impressed with awk speed.

> As far as I can see "join" will need a matching keyword in both tables.

The problem is (actually, was) that my database had three key variables. The 
solution was to use awk for creating a variable that is a string 
concatenation of the three variables. Then, the three variables became one, 
and I used join to merge the files. I'll add to the manual tips on how to use 
join, and then, everything that I once knew how to do with a slow dinosaur 
now I know how to do with the (much faster and free) statist and friends, 
sort, awk, join, etc...

Best,

Jakson

---

From: Bernhard Reiter <[email-masked]>
Date: Mon, 15 Aug 2005 13:30:39 +0200

Hi Jakson,

> In this case, it's just a matter of adding one more "help": {"--help", &help}, 
I have not checked the code, but it is probably as simple as that.

> -s --silent

we should not use abbreviation letters, because the unix convention is
that -sh would be the same as -s -h and we still allow the -silent
usage for backwards compatibility.
So for a long time (even after removing the "-long" options,
we would not be able to use short one, because there is the danger of 
old scripts.

> (if it's not there) in the library.

It is pretty dated, but shows the design philosphy nicely.
As I said I would not do some things as they describe there,
but it was a nice read. :)

> Your reasons are strong. I only regret that I lost my time
> developing recode.c and you lost yours translating its verbose messages. 

Well, this cannot be helped.
It needed the additional thoughs to come to the conclusion.

> and then, everything that I once knew how to do with a slow dinosaur 

Cool. :)
	Bernhard