LZ: Extended mem. format

[Prev][Next][Index][Thread]

LZ: Extended mem. format

------------------------------------------------------------------
  A proposed memory format for the RAM expander (v1.0, Sep. '96)
==================================================================


-------------
What is this?
=============


 This is my proposal of an extended-memory format for the TI-85/92 Ram
Expander (recently designed by Mel Tsai).  Please note that it is ONLY a
proposal.  It is subject to change as people make suggestions, or even to
be ignored entirely :(.
 I'd like to see a lot of high-end programs (games :) be developed for 
the TI, but it can't happen unless everyone wants it, so help out!
 And, of course, so far the RAM expander is only in beta, after all.  
But, if and when we have a working RAM expander, we'll need some way to 
ke ep data organized.  Hence, this proposal.


 This document deals only lightly with software implementation of the 
format described, because the format will likely undergo many changes 
before being finalized.


 There are a few (very few) things in here concerning the actual
hardware of the RAM expander, most notably the issue of how we know what
size the chip is.


 Besides working out the kinks in this spec / selecting
another spec, what we reall y need is some memory-management software /
library functions to handle memory allocation and transferring files /
variables between the TI and the expander. 
Volunteers?




---------
Contents:
=========
 A. Why we need a standard format for extended RAM
   1. How extended memory works
   2. Why we can't access it normally
   3. The uses of extended memory
   4. Necessity of doing this NOW
 B. Some considerations involving design of such a format
   1. On types of data
   2. On telling files apart
   3. On the size of memory
   4. On dynamic variables (files)
 C. The proposed format - the annoying little details
   1. Map of extended memory
 D. Where to now?










---------------------------------------------------
A. Why we need a standard forma t for extended RAM:
===================================================




-----------------------------
1. How extended memory works:
-----------------------------


 Extended memory on the 85/92 can work in one of two ways.
(The 82 can't be programmed in assembler and thus this is _probably_ 
possible only on the 85+.)


Either


  a. Someone figures out how to modify or extend the 85's
    internal memory manager to address more RAM


or


  b. Some device connects to the calculator m(probably via the
    link port), and will store and retrieve data on request.


 The first is probably impossible.  Even if it worked, we'd probably have 
to write special software to address the extra RAM.  At any rate, Mel 
Tsai's RAM expander works in the latter manner, and we have to work with 
what we have (the first would entail opening and modifying the calculator 
in any case, a risky business and probably not something us beginners 
should try :).
 




-----------------------------------
2. Why we can't access it normally:
-----------------------------------


 With an external device connected to the calculator, the TI cannot be 
set to treat it as its own RAM (at least not without some heavy-duty 
modification - almost as easy to make your own calculator!).  Thus 
extended memory must be read/written by routines we
write.






-------------------------------
3. The uses of extended memory:
-------------------------------




 Extended memory will probably be used by many programs for many 
purposes.  Games might store extra levels/sprites/sounds and download 
them as needed (We could have some real side-scrollers at last!).  
Perhaps a MIDI player would be stretching it, but then again, perhaps not 
(Did you say something about "Square waves"? We could always make an
add-on).


 As a side note, be aware that the TI-85 is just as powerful as a Gameboy -
more so, actually, because it's expandable - even to using the same 
processor (the Z80, albeit the GB has a custom one with a few more
opcodes), with the three exceptions of screen size, sound output
capabilities, and, of course, the 32k RAM capacity.
 (Plus you can get an adapter and use your TV to play gameboy games in 
color :( )


 The TI-92, of course, is significantly more powerful than a gameboy and 
is probably almost as powerful as some low-end desktop computers, 
except, again, for that tiny little RAM problem (tiny little RAM, indeed 
:).


 At any rate it is clear we will need to accomodate data of a variety of 
types from a variety of applications.  Without some sort of a standar d 
format (and possibly a standard function library), everyone will be 
stepping on each other's toes trying to get their fair share of RAM and 
not confuse it with everyone elses'.






-------------------------------
4. Necessity of doing this NOW:
-------------------------------


 If we wait until after some applications have been developed using 
extended memory, it will be very hard to make changes while maintaining 
compatablility with those applications.
 Plus, of course, if we wait too long, people may lose interest.








---------------------------------------------------------
B. Some considerations involving design of such a format:
=========================================================


--------------------
1. On types of data:
--------------------


 It is likely that there will be some types of data which it will be 
desireable to share between programs.  There will likely be sounds,
graphics, precompiled function libraries (High time!  #include files just
won't cut it for sound routines!  Or, for th at matter, for 
extended-memory access.), and probably a whole slew of more specialized 
data types.
 Any such format must allow, therefore, for data to be associated with a 
data type, and it must be possible for programmers to create and use 
their own data types.
 Keeping track of the data types that exist shouldn't be too much
trouble, now that everything important is archived at ticalc.org -
programmers need only send technical data to list-zshell to make everyone
aware of it.  Already, with the discover y that grayscale and sound are
possible on a TI, ticalc.org is proving it's
worth.






--------------------------
2. On telling files apart:
--------------------------


 It must be possible for any given program to find it's own data quickly 
(i.e. without checking 512k of sprites to see if they're what it's 
looking for - and how to tell, I don't know).  Obviously this will be 
inappropriate for files intended for general use rather than for use with 
a specific program.
 There are two ways I can see to handle this (if anyone has any more, it 
would be appreciated).


 a. Have program-specific data stored as type "raw" or somesuch, all in one
    big lump with the program's name on it.


 b. Have file-creator tags associated with each data item.
    Have a "general" creator tag as well, for files which aren't
    some program's internal data.


 In this proposal, I have chosen option b., because I consider it
desireable to be able to use standard programs to edit/play/view/whatever
data a program may use (Picture being able to replace a game's graphic set
and levels effortlessly, creating everyting right on the calculator.  We'd
get a lot more such contributions, dont'cha think?).
 At the same time, it is necessary to have data which is private to a 
program (high scores, passwords...), so I have included the "raw" 
data-type
tag of option a. as well.










-------------------------
3. On the size of memory:
-------------------------


 Since we do not know what size the extended memory will be, the format 
must be size-independant, for the most part.
 This is pretty obvious, but one issue does come up: how does the TI
know what size the chip is?
 I propose making the first byte on the chip represent the number of 
address bits the chip uses.  Thus there is no possibility of mucking up 
the circuit by trying to read from a non-existant location.
 Each program should begin by reading this byte. This has the added bonus
of telling us if there's a chip (or expander) there at all: if there isn't,
the byte will read zero!
 (Would this work with the hardware as-is, or would trying to read without
a chip muck it up?  Can someone please try to figure it out?)








--------------------------------
4. On dynamic variables (files):
--------------------------------
 We must have some way to deal with variable resizing. Specifically, do we
let everything bump into everything else and get fragmentation on a hard
disk?  That would tend to get pretty inefficient, wouldn't it? Or do we do
what the TI does internally (The 85, at least; I can't speak for the
others), and re-arrange things as needed so there's no leftover space? (I'm
using the convention of refering to "files" rather than "variables" because
it makes this easier to read.  Also, after all, who's going to store
'REAL's on this anyway?)


 Either one entails dynamically reallocating memory. This shouldn't be so
bad provided memory is ONLY rearranged at WELL-DEFINED times - times which
wouldn't usually be while a program is running, unless it's some sort of a
memory-management program.
 Specifically, I suggest giving each file its own physical and logical
size, just like a desktop computer.  The physical size would be some
appropriate amount, while the logical size would be the amount actually
used.
 "Raw" data could have the logical size equal the physical
size so the program doesn't have to waste time recalculating.  This
shouldn't be too much trouble, because few programs will really need to
keep creating and destroying files constantly in any event.


 Why have a "physical" size at all, then?  Because many file-formats won't
have a built-in indicator of how long a file is (text files, for example),
and also because we don't want to wind up sending empty bytes via a link
unnecessarily.


 Of course, this is all assuming we have some sort of library.  I think
this would be a good idea.  We could have one program that's not really a
program at all, but merely a set of routines (either at fixed offsets or
the first few bytes are a look-up table) that can be called to
create/destroy/reallocate file space.  Actually we'd need to be able to
format it, too.  We don't have to do a "hard format", though - just zap the
header info. Perhaps this library could be built into a future version of
Zshell?
 As long as we're at it, the library should create a "raw" file for dynamic
memory allocation - or would access be to slow to be worth it?  Just
thinking out loud here.  This is fast going beyond the scope of this do
cument, so I'll stop.


 In this proposal, I suggest a table at a fixed offset containing for each
file the filename, data-type and creator tags (4 bytes each, which permits
them to be nice readable ASCII text), and filesize.  Filesize is stored as
the number of contigious blocks (arbitrarily, 512 bytes), and the offset
within the last block to the physical EOF.
 Such a table is useful because it permits us to quickly find, insert, and
delete files.  It also puts in one place the vital information which is
used to "calculatorize" the files if/when they're downloaded to the TI's
internal RAM.  This may not be the best way to do it, but it doesn't permit
fragmenting of either files or free space.






-----------------------------------------------------
C. The proposed format - the annoying little details:
=====================================================
 After all that, the proposed format is remarkably simple!  Sorry - I just
wanted to anticipate as many issues as possible.
 Read on!


--------------------------
1. Map of extended memory:
--------------------------


offset 		contents 								type
------ 		--------								----
General information:
 $00		# of address bits						byte
 $01		version of standard ($00) 				byte
 $02		address of FDT							short int (2 bytes)
 $05		address of first entry in FDT			short int
 $07		offset to first FDT entry				short int
 		file block (need not be a multiple of 512)
 
File data table (FDT):
(Well, what do you call it?  It's more than just allocation...)
 $08		# of entries (files)					short int
 			That, trivia fans, means a max of
 			2^16 or 65,536 files. Enough for you?
 $09+		First entry in FDT
 
FDT entries:
 $xx		Data Type Tag							char[4]
 	VALUES:
 	 (Names with no uppercase letters are reserved for
 	  use in future versions of this format; all others
 	  are available to programmers.)
 		"raw " - Raw/internal data
 		"tiXX" - TI built-in formats; replace XX with
 				the ASCII values of the hex values for
 				the data type (see "varstor.txt"), all
 				letters in lowercase.
 				 i.e., the calculator's type"Strng" would
 				be "ti0c".
 
 		"prgm" - Program; it is preferred to make programs
 				of this type rather than of type "ti0c",
 				to indicate that they are programs and not
 				merely string variables.
 		
 		"Text" - Standard ASCII text data, with no EOF
 				character.


 		"Pic " - Picture file, in the following format:
 			$00		Number of bits (1 or 8)			byte
 			$01		Height (full-screen is 63)		byte
 			$02		Width (full-screen is 128)		byte
 			$03+	Data, raw, no padding, scanning
 				   first horizontal then vertical.
 					The minimum value (0) is white
 				   and the maximum (1 or 255) is
 				   black.
 		
 		"Pic2" - Enhanced picture file.
 			Same format as above, but with 2 bitplanes
 		   (After the "data" section is another one in
 		   the same format) - the second for transparency.
 		    0 is opaque and 255 is completely transparent.
 		    Anyone have ideas for future enhancements?
 		
 
 $xx+$04	Creator Tag								char[4]
 	VALUES:
 	 (The same names are reserved here as above.)
 	  	 "gnrl" - Gemeral-Ownership
 
 $xx+$08	Name									Pascal string
 		(Up to 20 bytes / 19 characters)		 (1st byte = length)
 		
 $yy		Numblocks								short int
		NOTE: We might save a few bytes by resizing
		this depending on chip size, but we'd lose
		in speed.
		
 $yy+$04	EOF offset					 			short int
 		(Physical EOF - (Logical EOF - 512))
 		In plain English, the number of physical
 		bytes in the last block.  Padding with 0s
 		up to 32 bytes n/a
 		
 	(repeat ad infinitum, or until we run out of files)


File data:
 $xx Start of first file
...








----------------
D. Where to now?
================
 This standard may or may not prove adequate to the task; at any rate
there is no way to test it until we have an actual working RAM expander.


 We need libraries to handle file management, and programs to transfer
between the expander and the calculator.
 We also need formats for various file types (i.e., compressed image,
sprite, grayscale sprite, etc. ought to do well for starters).


 If we can make this work, just imagine the possibilities!


====================================================================
 This document and its contents are Copyright 1996 by Daniel Knapp.
--------------------------------------------------------------------




                     /***********************************
                     *           Daniel Knapp           *
                     *       FeatherWate Software       *
                     *     email: dankna@bergen.org     *
                     *                                  *
                     *      Whose account this is,      *
                     *         I think I know.          *
                     * The sysop is in Berkely, though. *
                     * She will not see me typing here, *
                     *        To watch this Mac         *
                     *          Refuse to go.           *
                     ***********************************/