Work continues on the BASIC assembler enhancements. After spending some time learning how to use ObjAsm and the ROOL build tools effectively, the past week I've been suffering with a virus (of the human kind) which has slowed development to a crawl.
So far I've created a new optimised and assembler friendly version of all the pattern and lookup tables via ObjAsm macros in an automated fashion from the BASIC code. This makes it easy to update them as I simply re-run the BASIC application. The pains with the existing Acorn BASIC source code persist, so the plan is to add a branch from the existing code when a potential instruction begins with a "V" and branch back if no valid instruction is found. All the new code is in several new source files to avoid creating problems in the existing code. What remains then is to copy some code for storing the instruction into the correct location, and to figure out expression parsing to support the assembler syntax / notation discussed in the ROOL forums recently.
Unfortunately life is taking over for a while and it's going to slow down development. It make take a little more time than expected get this one finished but it remains the highest priority task we are working on.
Thursday, 23 June 2011
Tuesday, 14 June 2011
BASIC frustration
Another week has gone and this one has been quite frustrating. While some assistance from the ROOL forum's sorted out some bugs in the assembler (and I even found one in !DecAOF so it's not all my fault!), work on the BASIC source hasn't really progressed.
The BASIC source code is like a bowl of noodles - it's all tangled up and it's hard to see where something stops and something else starts. It reminds me of a BASIC program full of GOTO statements where you can't follow the flow, but in assembler it's worse as there is stack and flag management plus register preservation to consider. Reading this code is not only draining my enthusiasm it's also making me think seriously about a re-write. When you only have a few hours a day to work on something you don't need to waste most of that time trying to figure out what a bunch of crazy branch and magic numbers operations are trying to do. The memory management isn't suitable for what we need in TAG either, and it's missing key data-types as previously posted.
We've already internally had an informal correspondence about a JIT compiler written in a a higher level language instead of an interpreter written in assembler. I'm mulling over the consequences of all this at the moment while trying not to get any more stressed about the whole thing. Time for a deep breath...
The BASIC source code is like a bowl of noodles - it's all tangled up and it's hard to see where something stops and something else starts. It reminds me of a BASIC program full of GOTO statements where you can't follow the flow, but in assembler it's worse as there is stack and flag management plus register preservation to consider. Reading this code is not only draining my enthusiasm it's also making me think seriously about a re-write. When you only have a few hours a day to work on something you don't need to waste most of that time trying to figure out what a bunch of crazy branch and magic numbers operations are trying to do. The memory management isn't suitable for what we need in TAG either, and it's missing key data-types as previously posted.
We've already internally had an informal correspondence about a JIT compiler written in a a higher level language instead of an interpreter written in assembler. I'm mulling over the consequences of all this at the moment while trying not to get any more stressed about the whole thing. Time for a deep breath...
Sunday, 5 June 2011
Handlers and BBC BASIC
As I work through creating a test script for all these VFP and SIMD instructions, my mind is wandering a bit to one of the next challenges for TAG. TAG uses what we call Handlers to develop code for an application. The handlers respond to events thrown by TAG for all the useful things you might need in a game. These go from starting up and closing down, to rendering, and collisions. The events are comprehensive enough to deal with all the major things a game such as BHP requires.
Handlers are developed in assembler through a support library linked into the BASIC assembler. They are saved as a simple binary file with two entry points at the start, similar to a RISC OS module, one for frame events, and one for IRQ events. For us technical developers this is great, but for most people this is beyond their capabilities.
We considered writing a C interface, but ultimately for the average-Joe this isn't really much easier than writing in ARM assembler. You also need a whole load of compilers and stuff to get started and the Acorn PRM documentation is all aimed at assembly programming.
Acorn computers were always the easiest to use and to program. BBC BASIC was famous for it's no fuss approach to programming, and for being interpreted - making it much easier to understand what was going on. However over the years the language has become outdated. It lacks object orientation and Unicode, plus it can not be used outside of an "application" to create Modules or class libraries, nor can it easily be used within another application such as TAG.
Visual BASIC .NET has become the ultimate version of BASIC available today as it is a complete language. You can do everything that you can do in C/ C++/C# with a much more friendly syntax. My development teams of nearly 20 people created vast web applications, data-processing systems, and real-time shop floor manufacturing systems with it. Recently a customer who is into mathematics was dabbling in some programming, and no prizes for guessing he was using VB to write some code. I asked him about C/C++ and he said it was far to much hassle for what he wanted to do and BASIC was much simpler to use.
Sticking to the Acorn roots I'm contemplating a new version of BBC BASIC. Simple, interpreted, and as easy to use as always, but modern and powerful in it's capabilities. Backwards compatible with existing code, but forward thinking as it can be used to develop anything. Pulling the best bits of VB.NET as a language and syntax it would allow a framework for creating a class library that interfaces with RISC OS, Toolbox, and all the other things required for building applications rapidly.
A debugger is essential and the interpreter could be written with one in mind. An interpreter makes it possible to change the code in-line whilst debugging rather than having to have vastly complex compiler technology.
Perhaps it's biting off too much, I'm not sure. All I know is that it's taken a week part time to write an assembler from scratch using BBC BASIC and I'm back in love with the language that started off computing for me in 1984 on the BBC B. It would be great for RISC OS to maintain the Acorn tradition of being the easiest to use and program operating system bar none.
Handlers are developed in assembler through a support library linked into the BASIC assembler. They are saved as a simple binary file with two entry points at the start, similar to a RISC OS module, one for frame events, and one for IRQ events. For us technical developers this is great, but for most people this is beyond their capabilities.
We considered writing a C interface, but ultimately for the average-Joe this isn't really much easier than writing in ARM assembler. You also need a whole load of compilers and stuff to get started and the Acorn PRM documentation is all aimed at assembly programming.
Acorn computers were always the easiest to use and to program. BBC BASIC was famous for it's no fuss approach to programming, and for being interpreted - making it much easier to understand what was going on. However over the years the language has become outdated. It lacks object orientation and Unicode, plus it can not be used outside of an "application" to create Modules or class libraries, nor can it easily be used within another application such as TAG.
Visual BASIC .NET has become the ultimate version of BASIC available today as it is a complete language. You can do everything that you can do in C/ C++/C# with a much more friendly syntax. My development teams of nearly 20 people created vast web applications, data-processing systems, and real-time shop floor manufacturing systems with it. Recently a customer who is into mathematics was dabbling in some programming, and no prizes for guessing he was using VB to write some code. I asked him about C/C++ and he said it was far to much hassle for what he wanted to do and BASIC was much simpler to use.
Sticking to the Acorn roots I'm contemplating a new version of BBC BASIC. Simple, interpreted, and as easy to use as always, but modern and powerful in it's capabilities. Backwards compatible with existing code, but forward thinking as it can be used to develop anything. Pulling the best bits of VB.NET as a language and syntax it would allow a framework for creating a class library that interfaces with RISC OS, Toolbox, and all the other things required for building applications rapidly.
A debugger is essential and the interpreter could be written with one in mind. An interpreter makes it possible to change the code in-line whilst debugging rather than having to have vastly complex compiler technology.
Perhaps it's biting off too much, I'm not sure. All I know is that it's taken a week part time to write an assembler from scratch using BBC BASIC and I'm back in love with the language that started off computing for me in 1984 on the BBC B. It would be great for RISC OS to maintain the Acorn tradition of being the easiest to use and program operating system bar none.
Friday, 3 June 2011
VFPv3/SIMD Assembler Progressing
The last couple of days have been spent full-time on the VFPv3 / SIMD assembler for BASIC. It's now approaching a first complete version as I've been through a lot of tidying up and commenting. Some SIMD instructions remain to add as it's a very large instruction set but all the major ones are there now.
The exercise in writing an assembler has been fantastic in getting an understanding of the instructions and their uses. I've had to read through the variations of each instruction and the associated encoding formats. The only headache has been errors and inconsistencies in the ARM reference manual. I guess with everyone using C compilers these days few people delve into raw machine code.
The reason for all this work on VFP is to create a new 3D calculations library for TAG. This has to run using hardware floating point to achieve the performance and accuracy we require. We've decided not to go with Mesa Open GL now as it won't run fast enough and will take significant effort to get running. However we will revisit it in future if time permits. An open source driver for the PowerVR/SGX hardware may well end up in the Mesa code-base.
Today I also downloaded the RISC OS source and took a look at the BASIC source code. The idea is to add the VFP and SIMD instructions to the BASIC assembler while learning about how it works to add language extensions. Sadly the code appears poorly structured and documented. Thankfully it isn't that large so it should be possible to digest what is going on. I'll try and get my head around it after I've completed the VFP library.
The exercise in writing an assembler has been fantastic in getting an understanding of the instructions and their uses. I've had to read through the variations of each instruction and the associated encoding formats. The only headache has been errors and inconsistencies in the ARM reference manual. I guess with everyone using C compilers these days few people delve into raw machine code.
The reason for all this work on VFP is to create a new 3D calculations library for TAG. This has to run using hardware floating point to achieve the performance and accuracy we require. We've decided not to go with Mesa Open GL now as it won't run fast enough and will take significant effort to get running. However we will revisit it in future if time permits. An open source driver for the PowerVR/SGX hardware may well end up in the Mesa code-base.
Today I also downloaded the RISC OS source and took a look at the BASIC source code. The idea is to add the VFP and SIMD instructions to the BASIC assembler while learning about how it works to add language extensions. Sadly the code appears poorly structured and documented. Thankfully it isn't that large so it should be possible to digest what is going on. I'll try and get my head around it after I've completed the VFP library.
Monday, 30 May 2011
BASIC VFP and a decision on OpenGL
The re-write of TAG as TAG32 continues. Having completed a load of code relating to the structure of everything, we move on to the 3D library. There are two options on the table:
1) Write a new version of the TAG calculation and rendering code based on VFPv3 and SIMD. This will bring maximum performance. All the various methods would be exposed through the TAG32 SWI calls. This method fits with the RISC OS legacy in that there is assembler code at the core of the operating system. The down-side is a lack of standards implementation, making it RISC OS only code. That said the Windows/Airplay version of TAG wouldn't require this code, so it's not necessarily that much of a limitation.
2) Use OpenGL as the 3D library. This requires reviving the port of Mesa, and may open up hardware acceleration in the future. While this is an attractive option in many ways, the problem is a lack of performance. In order to squeeze high quality 3D graphics in a higher resolutions and colour depths out of the Cortex A8 it is essential to optimise the code. The Mesa code from the old RISC OS port is C code full of floating point and non-optimised functions and methods.
For now I've started work on adding VFPv3 and SIMD into the BASIC assembler. The ARM 'ARM' manual is a bit tricky to read, but after a couple of days of studying I now understand VFPv3 and at least some of the ideas behind SIMD.
The first version is a BASIC library which implements FNvfp("instruction") to assemble the instruction. All of the logic is based on lookup tables and a small amount of string processing code so it can be incorporated into the BASIC source in assembler at a later date. It uses EVAL so that expressions can be used with variables as with other instructions.
So far I've tested VMOV, VCVT, VSQRT, VDIV, and VMRS/VMSR and it works very nicely.
The VFPSupport module in RISC OS 5.17 is used to first make a context and enable VFP/SIMD operation as it's switched off by default.
I hope to put up a getting started with VFPv3 post in due course once I've completed work on the library and carried out some more testing.
1) Write a new version of the TAG calculation and rendering code based on VFPv3 and SIMD. This will bring maximum performance. All the various methods would be exposed through the TAG32 SWI calls. This method fits with the RISC OS legacy in that there is assembler code at the core of the operating system. The down-side is a lack of standards implementation, making it RISC OS only code. That said the Windows/Airplay version of TAG wouldn't require this code, so it's not necessarily that much of a limitation.
2) Use OpenGL as the 3D library. This requires reviving the port of Mesa, and may open up hardware acceleration in the future. While this is an attractive option in many ways, the problem is a lack of performance. In order to squeeze high quality 3D graphics in a higher resolutions and colour depths out of the Cortex A8 it is essential to optimise the code. The Mesa code from the old RISC OS port is C code full of floating point and non-optimised functions and methods.
For now I've started work on adding VFPv3 and SIMD into the BASIC assembler. The ARM 'ARM' manual is a bit tricky to read, but after a couple of days of studying I now understand VFPv3 and at least some of the ideas behind SIMD.
The first version is a BASIC library which implements FNvfp("instruction") to assemble the instruction. All of the logic is based on lookup tables and a small amount of string processing code so it can be incorporated into the BASIC source in assembler at a later date. It uses EVAL so that expressions can be used with variables as with other instructions.
So far I've tested VMOV, VCVT, VSQRT, VDIV, and VMRS/VMSR and it works very nicely.
The VFPSupport module in RISC OS 5.17 is used to first make a context and enable VFP/SIMD operation as it's switched off by default.
I hope to put up a getting started with VFPv3 post in due course once I've completed work on the library and carried out some more testing.
Wednesday, 25 May 2011
Threading and TBAFS64
TBA Software is not all about TAG and I've been reading through Forum posts and giving some thought to two areas in particular. Threading and a 64bit file system.
Pre-emptive multi-tasking
A few people have had a go at implementing threads over the years but all have been an add on, rather than being in the OS code. To make it work properly I think it has to be in the task control system of RISC OS, and it has to provide backwards compatibility for old code.
A new type of task is needed that is managed by a new process manager that has the common process, thread, local storage and synchronisation objects that are present in other systems such as Win32. My idea is that a single threaded process would be created to deal with all the co-operative stuff, and communicate with the "new world" through synchronisation objects. The SWI despatcher would need to understand what is thread safe and what isn't, and block if more than one call is pending to non 'safe' code (from outside the co-op thread). Modules / SWI calls would need a flag to say they are thread safe. WIMP poll would glue into the new process management. Memory protection should be implemented as standard, even if in a basic form, so that thread safe code expects memory protection from the outset.
There is of course a lot more to it technically than this, but I think it important to get a structure in place before more effort is spent on code that ultimately doesn't achieve what is required.
TBAFS64
Following a short discussion on the ROOL forums I have drafted a spec for a 64bit version of TBAFS which I've dubbed TBAFS64. TBAFS is an image file system so requires 64bit enhancements to Fileswitch, Filecore, and ADFS, before it could be implemented. However it could provide the new disc format that RISC OS needs to handle large disc - the Filecore formats use some quite traditional methods and seem to require quite a large amount of RAM.
TBAFS quite simply turns disc space into a heap into which it allocates blocks. It uses a single layer indexed directory structure and free block structure, plus a tree-indexed file data structure. The tree index gives very high performance when random accessing compressed files as it only has to work with the blocks in question. When files are not compressed they are still treated in blocks for simplicity.
Rather than tying the TBAFS format to the hardware disc layout in some way (as has been traditional with some file systems) TBAFS simply uses caching to limit the amount of I/O that is required.
The TBAFS64 spec includes an option for a journal, and dual copies of all the index data at each end of the disc. The idea with the journal is to keep the file structures intact in the event of a power loss. All writes for a particular "operation" are comitted to the journal before being committed elsewhere, and are only marked as complete when the journal has been flushed. In the event a power loss the journal will still contain entries that still require committing to disc. Of course care has to be taken in the underlying hardware design to avoid lazy write back caches that can defeat the integrity of the journal. One design note is to provide an option to put the journal on a separate disc, greatly improving potential I/O throughput.
It is possible to extend the journal to provide fully transaction safe operations such as those used by a database. This requires the addition of "transaction begin" and "transaction commit / roll-back" calls into the TBAFS API. This would cause headaches with blocking (locking in database terms) if a second attempt was made to alter structure (typically file table data) already modified by an open transaction.
As it stands someone needs to look at the 64bit code for Fileswitch and Filecore before this can get off the ground. I might be brave enough to take a look if time allows over the coming months but a lot needs to happen before then! :-)
Steady Progress
TAG Development Update
Another week has gone. Time seems to slip away when undertaking large development work on a part-time basis. Progress on TAG32 continues. The code that provides structure is coming on well - applications are initialising, and starting and stopping. The IRQ handlers are up and running along with the generation of events.
A few headaches remain in the structure of the old code. Particularly the use of data storage within the code through the EQUx instructions. While this was common practice way back when, and is OK for static values, it's not the way things should be done for variables and is unsafe for future use of threading and data execution protection. Common register use is in place for the necessary structures - R12 is global data, R11 application handle, R10 package, R9 a live resource header. Having data in a heap within a dynamic area per application gives much better control of memory usage, reduces fragmentation, reduces the chances of pollution, ensures everything is killed when an application exits, and reduces the use of the RMA to an absolute minimum.
I've had a few questions about software development on RISC OS and have asked a few as well. StrongED is my preferred way to edit BASIC code. TAG32 is pure assembler as ARM is by far the easiest CPU to write assembler for, and RISC OS is designed for assembler code throughout. It takes a bit of effort but sometimes it's worth it and it is what makes RISC OS a bit different.
The PC by comparison is a much more complex instruction set and nothing in is aimed at assembler code any more with C#, VB.NET, and C++ used for almost everything. This makes development faster on Windows but in many cases there is a total lack of understanding of what is going on. Too many applications ignore the lazy garbage collector (one of the most stupid optimisations ever dreamt up) and create vast structures of objects and use hundreds of megabytes of RAM as a result.
RISC OS lacks a decent debugger so it's a case of structuring and commenting code, testing one thing at a time, using the console, and even reverting to reading code out to make sure it makes sense. The only frustration with debugging assembler is that you can trash the machine very easily so it's good that RISC OS reboots very quickly. BASIC gives a huge amount of power to develop functions, and I have for example functions to define structures and write the variable names and offsets into a file for debug assistance. Use of LIBRARY replicates include and each file has a procedure to initialise it.
Another week has gone. Time seems to slip away when undertaking large development work on a part-time basis. Progress on TAG32 continues. The code that provides structure is coming on well - applications are initialising, and starting and stopping. The IRQ handlers are up and running along with the generation of events.
A few headaches remain in the structure of the old code. Particularly the use of data storage within the code through the EQUx instructions. While this was common practice way back when, and is OK for static values, it's not the way things should be done for variables and is unsafe for future use of threading and data execution protection. Common register use is in place for the necessary structures - R12 is global data, R11 application handle, R10 package, R9 a live resource header. Having data in a heap within a dynamic area per application gives much better control of memory usage, reduces fragmentation, reduces the chances of pollution, ensures everything is killed when an application exits, and reduces the use of the RMA to an absolute minimum.
I've had a few questions about software development on RISC OS and have asked a few as well. StrongED is my preferred way to edit BASIC code. TAG32 is pure assembler as ARM is by far the easiest CPU to write assembler for, and RISC OS is designed for assembler code throughout. It takes a bit of effort but sometimes it's worth it and it is what makes RISC OS a bit different.
The PC by comparison is a much more complex instruction set and nothing in is aimed at assembler code any more with C#, VB.NET, and C++ used for almost everything. This makes development faster on Windows but in many cases there is a total lack of understanding of what is going on. Too many applications ignore the lazy garbage collector (one of the most stupid optimisations ever dreamt up) and create vast structures of objects and use hundreds of megabytes of RAM as a result.
RISC OS lacks a decent debugger so it's a case of structuring and commenting code, testing one thing at a time, using the console, and even reverting to reading code out to make sure it makes sense. The only frustration with debugging assembler is that you can trash the machine very easily so it's good that RISC OS reboots very quickly. BASIC gives a huge amount of power to develop functions, and I have for example functions to define structures and write the variable names and offsets into a file for debug assistance. Use of LIBRARY replicates include and each file has a procedure to initialise it.
 |
| Use of StrongED, BASIC structures and defines, and heavily commented code |
Subscribe to:
Posts (Atom)