Copybook To Code (part 2)

There were two primary criteria for the generated JavaScript object.  First and foremost it had to be fast, low CPU consumption and elapsed time.  I was not concerned with how long the code generation took (within reason) as long as the generated code performed well. Secondly,  it needed to be easy to use, something that makes sense in the JavaScript world.  I won’t bore you with the details on the dozens of different iterations I went through, but here are some of the highlights:

  • My first attempt was not to generate code at all but to dynamically process the XML generated by CB2XML at run time to access/build the COMMAREA.  Some of the XML generated by CB2XML could be very large (120K or more).  This performed very poorly and was just plain ugly.
  • I generated code for every field in the copybook (both setters and getters).  While this performed okay the generated code could be huge, two or three times the size of the copybook.
  • I created a JavaScript object that would hold the values and the COBOL field descriptions defined in the copybook.  Then I created routines that would convert them into and out copybook format just prior to or just after invoking the COBOL program based on the descriptions.

On everything I tried I had lots of problems with groups, overlays, OCCURS and especially OCCURS DEPENDING ON structures.

Based on a suggestion from Scott, I focused on creating a byte array that just described each field in the copybook, but did not store any of the data. Its data type, signed/unsigned, precision, scale, length, position, OCCURS, OCCURS DEPENDING ON and the fields relationship to its owning group could all be stored in as little as 24 bytes. To store the values, I created a string that was the length of the COMMAREA and I would “set” or “get” the values directly in the string.  The code for setting and getting the fields was static and the same for all copybooks; it just used the field description to convert the data into the proper format whenever you set or got a field.  It performed very well. . . for a while.

Performance was great for copybook objects as long as you didn’t update the data in the copybook very many times.  Update it a few hundred times and I saw performance start to drop; a few thousand updates and it was terrible.  Update it ten thousand times and the region would go SOS (Short On Storage).  I couldn’t figure out why or how the region could run out of storage simply changing a field thousands of times, even if it was the same field over and over.  If it was a large COMMAREA it would run out of storage very quickly, even if the field I was updating was only one byte.  It turns out the problem was my use of a string to hold the COMMAREA buffer.  Strings in JavaScript are immutable.  That means that strings in JavaScript can’t be changed.  When you perform an operation on a string that would change it a new string is created, and strings are assigned by reference not by value.  So every time I updated one byte in my 32767 length string a new 32k string was created.  If  I did 32 updates to my string I had 1MB of memory left in use.  Now garbage collection would clean that up eventually, but it could not keep up with lots of updates occurring on very large strings.  I needed a new data/object type.  Scott came to the rescue again with StringBuffer.

The StringBuffer constructor creates an object with a static piece of storage.  Through the use of its methods, I could update as many times as needed without causing additional storage to be allocated.  I updated my scripts to use StringBuffer and performance started to fall in line with my expectations.  I found other places where I had gotten sloppy with my use of strings (lots of unnecessary concatenations/manipulations); after I cleaned those up CopybookToCode was a regular rocket.

I had a bit of work to figure out how to process uploaded files (MIME multi-part) via HTTP to HB.js and how to return the generated code as a file, but now a customer can use a browser to upload a COBOL copybook (or copybooks if your input copybook is different from your output copybook) and get back generated JavaScript that completely describes the in storage layout of the copybook.  Any field can be properly fetched or updated by using a getter or setter.  OCCURS, nested OCCURS and OCCURS DEPENDING ON are all supported and you can use Group names as if they were character fields to get or set entire blocks of data.

On occasion I still work on CopybookToCode, adding new features or fixing minor issues. I am in the process of changing the way the generated files are returned.  Right now you have to click on each generated file you want to download; soon a zip file containing all of the files will automatically download at the conclusion of the generation (of course I told Scott we needed zip support in HB.js for a customer . . . and he bought it).

It occurs to me after writing all of this you may not have a clear picture of how all of this comes together, so one more blog post on CopybookToCode to draw it out for you.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s