[fpc-other] Stanford Pascal Compiler successfully ported to Windows, OS/2 and Linux

[Bernd Oppolzer]

Hello FPC list,

I would like to inform you, that I ported an improved version
of the Stanford Pascal compiler (a descendant of the Wirth P4
compiler) to Windows, OS/2 and Linux.

I improved this compiler, which comes from the IBM mainframe,
by adding several features that I needed and that I found useful,
in the past few months.

Then I started to write a P-Code interpreter on Windows, which
interprets the generated P-Code; while doing this, I discovered
some severe portability issues, which required additional changes
in the compiler, for example: branch tables which were based on
the EBCDIC char sets; difficulties with sets of char or
sets of subranges of chars and so on. For all those problems,
I found portable solutions, that is: extensions or changes
to the P-Code, that eliminated those dependencies on the
code page of the target machine.

Now the compiler runs with identical results on all the target
platforms, including IBM mainframe; the generated P-Code files
may be transferred freely between the platforms. Running them
on the different platforms will yield the same results.

On the mainframe, the P-Code is translated to machine instructions;
on all the other platforms, the P-Code is interpreted. A true
P-Code translator on the non-mainframe platforms may follow later.
Maybe the P-Code to machine code translater from the mainframe
can be used as a starting point to do this, we will see; it does
some significant optimizations which are IMO indipendent of the
target machine's machine code.

BTW: the P-Code interpreter is written in ANSI-C.

Because I am working with an emulated mainframe (Hercules),
the interpreted compiler on Windows is in fact faster than
the machine code based compiler on Hercules (which will be not true
on a real IBM machine, of course). The compiler compiles itself
in 2 to 3 seconds on both platforms (using a laptop running
Windows 10 or the Hercules emulator; the laptop is not brand new).

You find some stories about my efforts of the last years and
months on my web site:

http://bernd-oppolzer.de/job9.htm

There are still some parts of the language missing on Windows etc.,
for example binary files (files, which are not files of char),
and modules (that is, linking of seperately compiled program units).
This will be added in the next weeks and months.

If you want to know more about this project, feel free
to contact me offline.

Kind regards,
merry Christmas and a happy new year

Bernd


_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Bernd Oppolzer]

Hello Mark,

on several occasions, I looked what FPC does, when I extended the
Stanford compiler, for example when I added support for direct write
of scalars.

At one time I recall that I decided explicitly to take another direction;
that was when I allowed shorter string constants to be assigned to
longer ones, for example:

var x: array [1 .. 200] of char;

x := 'some string';

IIRC, FPC fills with hex zeroes, but I prefer blanks - the blank
representation
of the target system ... which is different on the target systems; this
should
show to some of the readers here which are not familiar with IBM mainframes
some of the difficulties I had to get the P-Code really portable ... all
chars in
the (character) P-Code file had to be converted to character constants; all
places where character A - for example - was represented as numeric 193
(which is EBCDIC 'A') had to be found and corrected. Even such places where
the reference to 193 was not recognized at first sight, that is: offsets in
branch tables and bit strings representing sets.

This changes had to be done in the first pass of the compiler, which
translates
the Pascal source to P-Code; the second pass (P-Code to machine
instructions)
and the P-Code interpreter had to be adjusted accordingly.

See more information (including Pascal and P-Code examples) at the web site
mentioned below.

Regarding your question: There are - of course - some more incompatibilities
between FPC and Stanford Pascal; first of all, Stanford Pascal, as of
now, is a
very small language, compared to FPC. My extensions of the last months are
motivated by my own needs and interests (and by my C and Pascal/VS
experience,
to some degree); I wanted before all a Pascal based language that works
the same on
Windows, OS/2, Linux and IBM Mainframe. I didn't want to compete with FPC,
not even in the long term. This looks impossible to me, from today's view.

Kind regards

Bernd

Post by Bernd Oppolzer
Hello FPC list,
I would like to inform you, that I ported an improved version
of the Stanford Pascal compiler (a descendant of the Wirth P4
compiler) to Windows, OS/2 and Linux.
I improved this compiler, which comes from the IBM mainframe,
by adding several features that I needed and that I found useful,
in the past few months.
You find some stories about my efforts of the last years and
http://bernd-oppolzer.de/job9.htm

Well done Bernd. I remember that you were having some difficulty with
character array compatibility with FPC, are any of those still a problem?

_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Bernd Oppolzer]

Post by Bernd Oppolzer
chars in
the (character) P-Code file had to be converted to character
constants; all
places where character A - for example - was represented as numeric 193
(which is EBCDIC 'A') had to be found and corrected. Even such places where
the reference to 193 was not recognized at first sight, that is: offsets in
branch tables and bit strings representing sets.

I think you've made creditable progress in a difficult area. What are
you doing about PRED() and SUCC() as applied to CHARs?
Anybody with any sort of interest in mainframes is going to have to
consider EBCDIC for quite some while, but unfortunately there are
still people who insist that it's flawless. One of our wiki pages has
somebody confirm that EBCDIC has ^, but he then goes on to admit that
it's not in all codepages...

Thank you.

I think about "portability" in a certain way; to make it clear:

of course it is possible to write programs that are not portable
using my "new" compiler.

You are mentioning SUCC and PRED with CHAR; that is a very cood example.
These functions are implemented based on the underlying character set;
that means, that SUCC('R') is not 'S' on EBCDIC, because there is a gap
between
'R' and 'S in the EBCDIC codepage (six other characters between 'R' and 'S'
which are not alphabetic).

This is a portability problem which appears at the source code level (!)
and cannot be healed by the compiler. It is the same with the C language,
and the sensible programmer has to deal with this, if he or she wants to
have his or her programs really portable.

My problems with the Stanford compiler were different; if the compiler
generates
code which will not run on a platform using a different code page,
because it generates
branch tables when implementing case statements that imply a certain
code page,
this is a big problem and has to be fixed. The compiler implementor has
to find a
representation (in the P-Code, in this case), which will work on every
platform, that is:
which is independent of the code base - and does not prevent the
optimizations
done by the later stages of the compiler.

Same goes for the bit string representation of sets of char; in this
case, the construction
of the bit string has to be deferred until the code page can be determined
(P-Code translation or interpretation time). On Windows etc., the P-Code
interpreter
"translates" the P-Code to an internal representation on startup, and
that's the
time when the "portable" representation of set constants (of char) are
translated
to the bit string representation. See my web site for details.

Regarding ^:

"my" compiler supports different representations for the pointer symbol,
and for other
critical symbols, too:

^ @ -> for the pointer symbol (I use -> most of the time)
[ (. (/ for arrays
{ (* /* for comments ("comment" is supported, too, for historic
reasons)

no problem with EBCDIC. I do the editing on Windows most of the time and
move the
files to Hercules using the socket reader.

BTW: you find the compiler sources and a history of the extensions that
I applied in the last
months (or years) on the web site, too.

Kind regards

Bernd

_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Bernd Oppolzer]

Post by Bernd Oppolzer
"my" compiler supports different representations for the pointer symbol,
and for other
[ (. (/ for arrays
{ (* /* for comments ("comment" is supported, too, for historic
reasons)

/* was the form used in the first edition of Wirth's description of
Pascal (might have been before Jensen was there to help out). However
digraph, it's used as the address-of operator in Turbo Pascal and its
successors.

I don't like @, too. What you told me about the address-of operator
in Turbo Pascal etc. is a very strong reason to get rid of @; I will
think about it.
I converted all relevant sources (including the compiler itself) to ->,
so this would be ok for me.

I have a function ADDR, which gets the address of every variable (not only
heap elements), automatic and static variables (which I added), too.
ADDR returns an untyped pointer, which is compatible
to every other pointer type, much the same as the constant NIL.
There is a function PTRCAST, which casts every pointer type
to this untyped pointer type. And functions PTRADD, PTRDIFF and so on ...
I borrowed some of the names from PL/1.

Maybe pandora's box, but I need this to do some of the more systems related
work. For example: I rewrote the LE storage management in Pascal and
made it
available to the Stanford Pascal runtime (new functions ALLOC and FREE);
this works perfectly with Hercules and VM; it still has to be tested
with Windows - Linux - OS/2.

I added /* to the list of allowable symbols for comments, because it's
what I knew
from PL/1 and C. Comments have to be terminated by the same symbol which
started them, and: they may be nested (as with FPC).

Kind regards

Bernd
_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Alexander Stohr]

Post by Bernd Oppolzer

Post by Bernd Oppolzer
"my" compiler supports different representations for the pointer symbol,
and for other
[ (. (/ for arrays
{ (* /* for comments ("comment" is supported, too, for historic
reasons)

/* was the form used in the first edition of Wirth's description of
Pascal (might have been before Jensen was there to help out).

I converted all relevant sources (including the compiler itself) to ->,
so this would be ok for me.

This is indeed a very well known C style.

Post by Bernd Oppolzer
PTRADD, PTRDIFF and so on ...

This is not C style but with a minor macro helper something like this is
quickly added to a typical C program.
The difference between two pointers is defined as the number of elements
of the pointer type.
The addition or sum of two pointers is missing a bit of a straight
forward logic for me.
Advanced coding style recommendations (e.g. MISRA standards) would tell
you not to use such pointer math
because such constructs have a higher chance to sometimes mislead the
performing programmer and thus would more likely lead up to codes with
functional bugs.
The first answer for such operators often is: use the right container
for the value you are in need and avoid all those operators.
And the second answer is: If you are really in the true rare need (e.g.
for system programming, ring buffers, atomic operations, code sizes +
register savings + extreme efficiency) then keep these codes and data
very isolted, very well documented and finally deeply tested for all
operating conditions.

Post by Bernd Oppolzer
Maybe pandora's box, but I need this to do some of the more systems related
work. For example: I rewrote the LE storage management in Pascal and
made it
available to the Stanford Pascal runtime (new functions ALLOC and FREE);
this works perfectly with Hercules and VM; it still has to be tested
with Windows - Linux - OS/2.

Is the your lower end of those module touching on a system API level for
that?
(e.g. the Linux ABI, OS/2 system peronality, WIN32 api - rather than
malloc()/free() as a C standard library provides it)

Post by Bernd Oppolzer
I added /* to the list of allowable symbols for comments, because it's
what I knew
from PL/1 and C. Comments have to be terminated by the same symbol which
started them, and: they may be nested (as with FPC).

From a C program portability view I would futher "tick" for the support
of the "//" comment operator as a single sided one.
On the other side, even for ages there is the option of pascal-2-c
converters (p2c) that do the other way around, but would have some
smaller problem if such a token was suddenly added. If sources of the
tool are available then someone that is in need for support could solve
it easily.

just my 2 euro cents.

regards and good christmas times.
-Alex.
_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Bernd Oppolzer]

Post by Alexander Stohr
The first answer for such operators often is: use the right container
for the value you are in need and avoid all those operators.

Bear in mind that this is very old code that Bernd is maintaining, and
long predates any concept of objects or of containers other than
Pascal records.

Post by Alexander Stohr
Is the your lower end of those module touching on a system API level for
that?
(e.g. the Linux ABI, OS/2 system peronality, WIN32 api - rather than
malloc()/free() as a C standard library provides it)

Again, bear in mind that one of his major targets is a late-70s
vintage IBM CMS, which predates most APIs as we know them (in fact,
long predates the terms API and ABI themselves :-)

Thank you for your feedback.

I'm aware of all the concerns you have.
That why I wrote about "pandora's box".
Indeed I'd prefer to use the pointer arithmetic facilities only in certain
restricted circumstances, when this seems the best way to go and all
other (more indirect) ways to get the work done are still more clumsy
and more difficult to understand.

Anyway, it's now possible - for example - to access an array by moving
a pointer over the elements, but I, too, believe, that that's not how it
should
be done, regularly. BTW, I had to remove some sort of self check from the
Stanford Pascal runtime; it checked (before), that pointers only pointed
to the valid range of heap addresses; because the "traditional" heap
consisted
of one contiguous segment, this was very easy to implement. But I had to
remove this check, because pointers now can point to auto variables,
static variables, "new" alloc areas etc. as well.

For your second question regarding the system API:

on the mainframe, the lower end of the LE storage management consists
of the GETMAIN and FREEMAIN macros; that is, the storage management
provided by the operating system. I provided for test purposes two
functions ALLOCX and FREEX which map directly to GETMAIN and FREEMAIN;
this worked, but was very slow. The LE storage management is optimized
for C and C++ and tries to support many small areas of storage, minimizes
fragmentation and optimizes performance. See this IBM presentation, which
is mirrored on my web site:

http://bernd-oppolzer.de/stackheap.pdf

The problem, why I had to add this to the Pascal runtime, was: the original
Stanford Pascal runtime only had functions new, mark, and release. Release
releases all storage which was required since the last mark call - but there
is no invidual "free" of areas. But I wanted to port an application to
Stanford
Pascal which required invidiual allocs and frees (like C malloc/free). I
decided
to add alloc/free and leave new/mark/release untouched for the moment,
because it is used in the compiler.

ALLOCX/FREEX goes directly to GETMAIN/FREEMAIN (which is slow), but
ALLOC/FREE implements the LE solution for malloc/free, which is
significantly faster.

The port of ALLOC/FREE to WIndows etc. still has to be done; maybe there
is no need to re-implement the LE solution, because the normal malloc/free
which is available to the interpreter may be acceptable regarding
performance;
so ALLOCX / FREEX could simply be the same as ALLOC / FREE on the PC.

You can find the source of PASLIBX, which implements the LE solution in
Pascal, here:

http://bernd-oppolzer.de/paslibx.pas

See resources paragraph of my Stanford Pascal related web site:

http://bernd-oppolzer.de/job9.htm

Feel free to ask if you have more questions or suggestions.

Kind regards
and good christmas times to you, too

Bernd

_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Alexander Stohr]

Post by Bernd Oppolzer
Thank you for your feedback.

Thank you for your kind andswers.

Post by Bernd Oppolzer
BTW, I had to remove some sort of self check from the
Stanford Pascal runtime; it checked (before), that pointers only pointed
to the valid range of heap addresses; because the "traditional" heap
consisted
of one contiguous segment, this was very easy to implement. But I had to
remove this check, because pointers now can point to auto variables,
static variables, "new" alloc areas etc. as well.

It would still work for some applications. But they would be few.
The vast majority of real use projects probably tend to break those box.
So if it was an option with an enable switch, most would disable it.
Probably for that reason it would never be worth keeping it at all.

Post by Bernd Oppolzer
The problem, why I had to add this to the Pascal runtime, was: the original
Stanford Pascal runtime only had functions new, mark, and release. Release
releases all storage which was required since the last mark call - but there
is no invidual "free" of areas. But I wanted to port an application to
Stanford
Pascal which required invidiual allocs and frees (like C malloc/free).
I decided
to add alloc/free and leave new/mark/release untouched for the moment,
because it is used in the compiler.

So it was created more like a stack allocator. The allocations were
local to the
functions or context they were allocated in and at some waypoint
(exit/return)
all of them were released. Thats not a generic universal heap design but
rather
goes to the level where gaps of sometimes named but then unused items
increase over time
and on persistent would need some garbage collection (time loss!) any
now and then.
But for that design as above growth and shrinkage is determined by the
code path.
Under some conditions the growth might be very determined
but the shrinkage is always very fixed whilst beeing much rarer.
(I feel a little similarity to older stack rewinding concepts in C
exception/resume features.)

I see you did wise to keep those items out of yournew codes for the project
whilst keeping it untouched for the moment in the existing codes that
dont interfere.

Do you see a good chance to use some larger existing code bases and test
suites for verifying the compiler?
Do you have some heap tracking functionality inside so that e.g. "1234
heap Bytes lost" it printed at exit?
Is there a some debug option for stack max size tracking?
Is there something for stack/heap object out of bounds writes/access?
(thinking of magic word fences in between, and of heap sanity checking)

Regards, Alex.

_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Bernd Oppolzer]

Post by Bernd Oppolzer
Thank you for your feedback.

Thank you for your kind answers.

You're welcome; I'm happy to meet someone who is interested in my work :-)

Post by Bernd Oppolzer
BTW, I had to remove some sort of self check from the
Stanford Pascal runtime; it checked (before), that pointers only pointed
to the valid range of heap addresses; because the "traditional" heap
consisted
of one contiguous segment, this was very easy to implement. But I had to
remove this check, because pointers now can point to auto variables,
static variables, "new" alloc areas etc. as well.

It would still work for some applications. But they would be few.
The vast majority of real use projects probably tend to break those box.
So if it was an option with an enable switch, most would disable it.
Probably for that reason it would never be worth keeping it at all.

The Pascal documents say that they implemented it in a very basic manner,
just to be sufficient for the needs of the compiler; and that future
implementors
are free to replace the storage management by more sophisticated solutions.
The heap elements allocated by the compiler are - for example - all
freed, when
a certain block is completely compiled, that is: the internal lists of
definitions
are freed, because they are no longer needed. I kept the mark/release
logic,
because I wanted to keep this logic. The new/mark/release areas are
completely
different from the alloc/free areas. The whole mark/release area is
allocated
at startup and cannot be enlarged (4 MB minus stack at the moment, can
be configured). The alloc/free area is allocated on an as-needed base in
64 k chunks;
limited to ca. 8 to 10 MB due to address range limitations - may grow much
larger, if 31-bit addressing were possible.

Post by Bernd Oppolzer
The problem, why I had to add this to the Pascal runtime, was: the original
Stanford Pascal runtime only had functions new, mark, and release. Release
releases all storage which was required since the last mark call - but there
is no invidual "free" of areas. But I wanted to port an application
to Stanford
Pascal which required invidiual allocs and frees (like C
malloc/free). I decided
to add alloc/free and leave new/mark/release untouched for the moment,
because it is used in the compiler.

So it was created more like a stack allocator. The allocations were
local to the
functions or context they were allocated in and at some waypoint
(exit/return)
all of them were released. Thats not a generic universal heap design
but rather
goes to the level where gaps of sometimes named but then unused items
increase over time
and on persistent would need some garbage collection (time loss!) any
now and then.
But for that design as above growth and shrinkage is determined by the
code path.
Under some conditions the growth might be very determined
but the shrinkage is always very fixed whilst beeing much rarer.
(I feel a little similarity to older stack rewinding concepts in C
exception/resume features.)
I see you did wise to keep those items out of yournew codes for the project
whilst keeping it untouched for the moment in the existing codes that
dont interfere.
Do you see a good chance to use some larger existing code bases and
test suites for verifying the compiler?
Do you have some heap tracking functionality inside so that e.g. "1234
heap Bytes lost" it printed at exit?
Is there a some debug option for stack max size tracking?
Is there something for stack/heap object out of bounds writes/access?
(thinking of magic word fences in between, and of heap sanity checking)

The first test for the compiler is always the compiler itself (first and
second pass);
it should compile itself again and again and yield the same results.
Then I collected
over the time some 30 testcases, which cover different areas;
esspecially the new
features that I added. I am a big fan of test driven development, so I
often added new
statements and features which first lead to a compiler error, and then I
implemented
them, until they worked as the should. Now these test cases are kept for
regression testing.

For the LE heap management, which is a sort of addendum to the Pascal
runtime
(the compiler doesn't need it):

there are functions that give statistics on heap usage at the end of the
process or
at any point in time in between

there are functions that check the heap for integrity (same checks as
suggested by
the IBM paper - the LE heap management technology is a product of IBM
Watson
Research Center, see the presentation link some days ago)

I wrote a program to check for memory leaks (in ANSI C), which works
with the
"normal" LE heap management (as provided by IBM); you call this program
twice
at different points in time, and the program tells you, which areas have
been
allocated and not freed in the meantime; this was very helpful when
finding memory
leaks for my customers (insurance companies with PL/1 code base ... this
tool and
the effort that I had to understand the LE heap management gave me the
idea to
rewrite it in Pascal) - I believe, it should be possible to rewrite that
in Pascal, too.

in fact, it took me some weeks and very much tests to get the heap
management functions
running without errors (hopefully); there are many special cases which
need special
treatment (when combining adjacent free areas etc. and maintaining the
trees of
free areas). This was one of the most difficult pieces of software I
wrote in the last years.

Stack and heap max size is no problem; the values are collected
regularly (and printed
at process end, if you wish).

There are magic words etc. and several security measures regarding
overwrite of
heap control blocks etc., but with normal processing, errors will be
discovered at
free processing, only. And: not all heap areas are regularly checked at
every alloc
and free request. The LE system has a HEAPCHK option, which does exactly
that
(check the entire heap at every request). If needed, it would be easy to
add this, because
the function to check the heap is already available.

If you want more information, see the IBM presentation:
http://bernd-oppolzer.de/stackheap.pdf
or the Source code of the Heap Management:
http://bernd-oppolzer.de/paslibx.pas
or: feel free to ask further questions.

Kind regards

Bernd

Regards, Alex.
_______________________________________________
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other

[Florian Klaempfl]

Post by Bernd Oppolzer
Hello FPC list,
I would like to inform you, that I ported an improved version
of the Stanford Pascal compiler (a descendant of the Wirth P4
compiler) to Windows, OS/2 and Linux.

Glad to hear that FPC is not the only oss/free pascal compiler being
developed :)

_______________________________________________
fpc-other maillist - fpc-***@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-other