How to encrypt sensitive data in web.config

As we write custom WebAPI applications, we need to take extra steps to secure sensitive information after deployment. Below is one of the steps you can take to encrypt the web.config file, which is a popular place where configurable sensitive information is stored.

Powerpoint slide

Cheers!

Writing records to disk immediately using MIO

Last week, I ran into a scenario where I needed to create two subroutines that are called sequentially by another subroutine. The first sub creates a record and then the second sub queries the database for the record and performs extra processing. The workflow looks like this:

Overall Subroutine Begins
   Subroutine 1: creates a record
   Subroutine 2: does a select to obtain the record created in subroutine 1
Overall Subroutine Ends

There was a problem with this workflow that I was using just a FOR_THE command to create the record in sub 1. Since the record created in sub 1 stays in the running buffer, it never gets written back to disk until the overall sub ends. Because of that, sub 2 never finds the new record.

To get around this, my colleagues at Ferrilli suggested manually changing the MIO level so that sub 1's record is written to disk immediately. The code to be added is:

CALL @MIO.TRANSACTION(MIO.TX.SUSPEND)
FOR_THE record_id
  * set field values
END_THE record_id
CALL @MIO.TRANSACTION(MIO.TX.RESUME)

The two MIO calls modify the level of the buffer and that's where the magic happens. Using MIO.TX.SUSPEND, in effect, temporarily starts a new process level at level 0.  All I/O that is performed after the call to MIO.TRANSACTION using MIO.TX.SUSPEND will be held in MIO's buffers until the call to MIO.TRANSACTION using MIO.TX.RESUME is reached. In our case, as MIO.TX.RESUME is reached, the buffer of the transaction is flushed and the record is written to disk. Afterward, the MIO level returns to the normal running state. So my pseudo code looks like this at the end:

Overall Subroutine Begins
   Subroutine 1: creates a record
      CALL @MIO.TRANSACTION(MIO.TX.SUSPEND)
      FOR_THE record_id
          * set field values
      END_THE record_id
      CALL @MIO.TRANSACTION(MIO.TX.RESUME)
   Subroutine 2: does a select to obtain the record created in subroutine 1
Overall Subroutine Ends

Thanks Neal, Lea, and Geoff for the documentation, and Thomas Mantooth for verifying the code. GO FIGTEAM!

Record markers to ASCII (and vice versa)

Credits to Thomas Mantooth.

The UniData delimiters are in reverse order of their hierarchy. So, going in the other direction...

@RM (Record mark) = CHAR(255)

@FM (Field mark) = CHAR(254)

@VM (Value mark) = CHAR(253)

@SM (Sub-value mark) = CHAR(252)

@TM (Text mark) = CHAR(251)

One neat trick you can make use of with this comes into play when you use the REMOVE command. It assigns a value that indicates what type of delimiter it returns. Most of the time, you don't care what the specific delimiter is - you're just looking to see if it's zero, which is returned at the end when there's no delimiter. However, if you're dealing with something that can have multiple delimiters, you can use REMOVE and the delimiter value to build a new dynamic array with the same delimiters very simply.

LOOP

    REMOVE X.VALUE FROM XL.ARRAY SETTING X.DELIMITER

    * Do something to manipulate the data

    XL.NEW.ARRAY := X.VALUE

UNTIL NOT(X,DELIM) DO

    XL.NEW.ARRAY := CHAR(256 - X.DELIMITER)

REPEAT

Check if a record exists

There are a few ways to check if a record exists in a file.

1. Call the S.VERIFY.RECORD.EXISTS subroutine:

X.RECORD.ID = record ID to check

CALL S.VERIFY.RECORD.EXISTS(X.EXISTS, X.FILE.NAME, X.RECORD.ID)

IF X.EXISTS = 0 THEN // record doesn't exists
END

2. Use  CONFIRMED command (this is good for when the subroutine can't be called, such as in IS sub):

X.FILE.NAME = A.FILE.NAME
X.RECORD.ID = A.RECORD.ID
A.EXISTS = 0
IF LEN(X.FILE.NAME) AND LEN(X.RECORD.ID) THEN
CONFIRM X.RECORD.ID IN_FILE(X.FILE.NAME)

IF CONFIRMED THEN
A.EXISTS = 1
END
END

Source: Neal Webb

Padding preceding zeros to Colleague ID

In Colleague, ID numbers often have preceding 0s, and these are often truncated as the data is imported into Colleague. To add a small check to the ID to make sure it always is 7 number-long, you can add the following:

* Add zeros padding to PERSON ID
X.PERSON.ID.LEN = LEN(A.PERSON.ID)
X.ZEROS.TO.PAD = 7 - X.PERSON.ID.LEN
IF (X.ZEROS.TO.PAD GT 0) THEN
   A.PERSON.ID = STR("0", X.ZEROS.TO.PAD) : A.PERSON.ID
END

Detail to screens cross applications

If you are familiar with the application tree in Colleague, you know we can detail (UI screen) from a process to another within the same environment, or up one level, but not to the adjunct application. The tree looks like this:

                        UT

                          |

                     CORE

                   /      |      \

               ST     HR     CA

A screen from ST can detail to another screen in ST, or CORE, or UT, but it can't go to HR or CA. There's a trick to make this happen. If you want to detail to a HR screen from ST, you can do this:

   CALL S.SESSION.INIT('HR')

   CALL_SCREEN SUSPEND HRS105($PRIMARY)

   CALL S.SESSION.INIT('ST')

This code would initialize all environment variables required from HR application, open the screen, then reinitialize the ST application. 

Credit:  Eric Small from eCommunities.

WebAPI connection exception Error Subset Found: Server error-00150-No response returned from the server

When writing a Colleague custom WebAPI, I encountered a connection error while executing a transaction:

[Ellucian.Data.Colleague.Exceptions.ColleagueTransactionException] = {"Error Subset Found: Server error-00150-No response returned from the server."}

I made sure my credential was correct and the connection was established before calling the transaction. After much debugging, I found out that the string that was passed to the transaction exceeded the size limit of a field in Colleague. I passed in a 15 character long string and tried to write it to a 10 character long field in Colleague through the transaction, and it resulted in a truncation error which ended the Colleague session. This also returned the error above. They do not look like they're related, so it was hard to track down. Make sure you check the size of the arguments of the transaction when you get these error.

Check if a record is locked

It is useful to check to see if the record is lock before beginning any processing. This eliminates the process to spin and wait for the record to be unlocked. When you work withe the WebAPI, it is more important to return a status of "resource is unavailable please try again" than to wait and risk the process timing out. To check to see if a record is locked, follow the below template:

XKV.FILE = V.BPV.DOC.IDS
XR.FILE = ""
CALL @MIO.READ.RECORD(MIO.READU.EXIT.ON.LOCKED,XFV.FILE,XKV.FILE,XR.FILE)
IF MIO.STATUS = MIO.STAT.LOCKED THEN ;* file is locked
   X.ERROR = 1
   * Output your message, set the status, or do some other process
END ELSE ;* record is not lock, proceed like normal
   * normal record read/write
END

The following give you different statuses of the lock check:

To read a record from a previously opened file.
@MIO.READ.RECORD(read_option, FV.FILE, KV.FILE, R.FILE)

Where FV.FILE is the previously opened file, KV.FILE is the key of the
record to be read, and R.FILE is the array into which the record will be
read.

Read_option is one of the following. These are equates so they should not
be surrounded by quotes.

MIO.READ - Reads the record without locking

MIO.READU - Reads the record and locks. If the record is already locked by
someone else, the user will get a message and will have the opportunity to
cancel the command.

MIO.READU.NO.CANCEL - Same as READU but without the option to cancel if the
record is locked.

MIO.READ.VERIFY - Checks to see if the record exists. R.FILE will be set to
1 if the record exists and 0 if the record does not exist. The entire
record is not read into the R.FILE variable if this option is used.

MIO.READ.ORIG - Will read the record from disk instead of from the MIO
memory buffers.

Modify/Compile Envision processes in the terminal

If you have access to Desktop UI's terminal, or connection to the unidata application layer, you can directly modify and compile envision processes. Altho this is not a recommended practice, it can come in handy when you need to put out fire quickly.

In the terminal, open the file you want to edit with command:

:AE ST.SUBROUTINES C70.S.MY.SUBROUTINE

You need to determine the location of the process you're trying to edit. If it's a FA subroutine, then then instead of ST.SUBROUTINES, use FA.SUBROUTINES.

You can search for the text using "/Text to search". After you find the line, to can go to that line by just typing in the line number. You can then replace the text with "C/"Text to be replaced"/"Text to replace with". After you are done editing, use the command "FIB" to File and Batch, which also compiles the process. Boom, you're done!

You can find more about AE editor here Unidata Command Refs

self service "Could not validate this request's antiforgery token"

You may encounter this error while customizing Ellucian's self service. To fix this error, make sure you run the application in Local IIS, instead of IIS Express

After setting the default to Local IIS, you will want to delete all the cookies pertaining to the localhost. In chrome, go to Settings - Content Settings - All cookies and site data... then search for cookies belong to localhost

Delete these and try again. Similar steps should be taken with the browser you use to test self service.

How to find references to a field in SQL backend

This query will give you a good list of reference on a particular field. You can find all references to it from processes and screens. This requires you to have a SQL backend and a studio to query the database. Replace 'APPLICATIONS.ID' with name of the field you want to find reference.

declare @fieldName as varchar(20) = 'APPLICATIONS.ID'
SELECT PROCESS_NAME as 'Process ID', PROCESS_TYPE as 'Process Type', PROCESS_DIRECT_ACCESS_NAME as 'UI Mnemonic' from PRCS_DEF A left JOIN PRCS_CTL B
ON A.PROCESS_NAME = B.PROCESS_MNEMONIC
where PROCESS_DATA_ELEMENTS like '%'+@fieldName+'%' or
PROCESS_DEMAND_ELEMENTS like '%'+@fieldName+'%' or PROCESS_DEMAND_POINTERS like '%'+@fieldName+'%'
ORDER BY PROCESS_DIRECT_ACCESS_NAME DESC, PROCESS_TYPE 

The result would look something like this:

Using ESEARCH to find hidden process references

Colleague Studio is a very user friendly tool that can do many things with simple mouse clicks. However, it's not perfect, and sometimes it does not do what it says it does 100% to completion. Let's take "Show Reference" for example. Colleague Studio will show you a partial list of processes that reference a particular process. If you still have access to the terminal in Desktop UI, using ESEARCH command can help in these kinds of research.

We will look at one of the Ellucian's delivered subroutine, S.CHECK.IF.WITHDRAWN, for this example. When checking for references in Colleague Studio, the list looks like this:

Now that I know this subroutine is mainly called in ST, I can go to the terminal and check for the complete list.

The above screen shows the usage of ESEARCH. You specify the folder/category of processes you want to search for. The above searches for UI/Webadvisor forms in ST. After the search, the result is saved in a savedlist named DTP. When you open DTP up, either in the terminal or in UI using SLED, you will see the more complete list of UI/Webadvisor forms that reference S.CHECK.IF.WITHDRAWN in ST.

Looking at the result, we can see more processes that reference S.CHECK.IF.WITHDRAWN than what Colleague Studio provides. I have found that Colleague doesn't not maintain references to custom processes very well, so ESEARCH is a good way to double-check. You can search for processes in corresponding applications:

appl.SUBROUTINES -> appl subroutines

appl.SOUCE -> appl screens

All of this requires you to have access to the terminal, of course. You can do a similar search using the window explorer on the Colleague file server, but that takes way longer for me.

Killing Unidata session (softly with STOPUDT)

A while back when Colleague UI was first introduced, users started to X'ing out of the web-browser to close the session instead of using the logout button. That resulted in many runaways processes that used up the small license pool we had. I learned to kill those sessions through the terminal as following:

Open Colleague Desktop UI terminal:

LISTUSER: this lists all active sessions

LISTUSER | find "username": this finds a particular user's active sessions with "username"

Kill the session by:

STOPUDT usrnbr(user number): kill the session using the user's number, it's the number on the second column after using LISTUSER

* Note: insert a bang ("!") in front of LISTUSER or STOPUDT to use admin right if the commands don't work

You can use LISTUSER to check to see if your process is still running or the screen has frozen. Many times I waited for a process for so long just to find out the session was disconnected.

Problem with sorting dates in Envision

I have encountered a bug today with one of the sort subroutines, S.SORT.BY.DL, that could result to other sorting issues. In one of the processes, the dates were not sorted in the correct descending-order. After some much digging through log files and test cases, I finally shred some lights to the issue. If the date array to be sorted contains date that is before 5/18/1995, the array will not be sorted correctly. The reason for this bug is related to the way Unidata stores date in the database. Unidata dates are stored as a counter from a default date, 1/1/1968. All dates are presented as a numbers, so 1/2/1968 has a value of 2, and 1/5/1968 has a value of 5 and so on. 

For that reason, let's consider this example. Say we have an array of dates of 01/01/2000 and 5/17/1995. In envision these two dates are presented by 11689 and 9999. Because Unibasic(Envision) is a loosely-typed language, these numbers can be considered as "strings" at run-time when their type is not strictly/explicitly enforced. So after sorting the date array with descending order:

XL.DATES = 11689:@VM:9999

You will get this:

XL.DATES = 9999:@VM:11689

This is because 9 is greater than 1, so 9999, as a string, is greater than string 11689. This is incorrect because 11689 is greater than 9999, as well as 01/01/2000 is later than 5/17/1995. I also tested S.SORT.ASSOC.DL and it seems to behave the same way. 

This issue is more applicable when you work with historical data. It can happen again in 80k days when the date reach 100,000. That is more than 200 years from now, I wonder if Unidata still exists, but I am sure I won't be around to worry about it.

How to get/query the list of files in a folder from Envision

Say you have a list of file in HOLD_SHARED_MYFOLDER directory in Colleague, and you want to read in the list of files in MYFOLDER and do something with them, maybe read or write or both.

What you need is read/write access to MYFOLDER. Your admin should know how to set it up, and a VOC entry should already be created for it. 

In Colleague Studio, use the following code to get back a list of files inside MYFOLDER directory:

X.STMT = 'SELECT HOLD_SHARED_MYFOLDER'
CALL S.EXECUTE(X.STMT)
XL.FILES.NAME = ''
CALL S.READLIST(XL.FILES.NAME,0,X.STATUS)
CONVERT @FM:@SM:@VM:@AM TO @VM:@VM:@VM:@VM IN XL.FILES.NAME

Now you will have all the files' names in MYFOLDER populated in XL.FILES.NAME. To open each file up, you iterate them with S.OPEN.SEQ subroutine

FOR X = 1 to DCOUNT(XL.FILES.NAME, @VM)
   * Open the file
   CALL S.OPEN.SEQ("HOLD_SHARED_MYFOLDER", XL.FILES.NAME<1,X>, X, "R", X.ERROR.MSG)
   * Read from the file
   CALL S.READ.SEQ(X.TEXT, X.EOF, X, X.ERROR, X.MSG)
   * Close the file
   CALL S.CLOSE.SEQ(X, X.ERROR, X.MSG)
NEXT X

Notes on S.EXECUTE subroutine

S.EXECUTE will execute a sentence to perform operations, such as a select statement against the database. This is Datatel's version of the Unibasic EXECUTE command (you can type HELP EXECUTE at the colon prompt for additional information about EXECUTE). Since EXECUTE only works for Unidata clients, we encourage people to use S.EXECUTE in its place.

UniQuery-syntax statements can be used with S.EXECUTE on any underlying database. These statements, when used with S.EXECUTE, will be converted by Datatel's underlying MIO components to the native database query language for executing the select statement.

There is only one argument for this subroutine:
A.STMT Input only The statement to execute; the format of this statement should be:
[<options>] <command> [<arguments>]

<options> are optional; it is a list of zero or more options that are separated by spaces, each of which begins with a dash ('-') character. Valid options are:

-H = Hush the output that may be generated by the command being executed; for example, the 'n records selected' message when a select statement is executed
-NM = Do not map SELECT/SSELECT commands to MIOSEL/MIOSELS; this option is NOT recommended for use unless you are certain the file being selected exists on the application server AND you are certain that the -NM option is necessary
-C = Capture output from the command in the CAPTURED.OUTPUT common variable
-DB = Force debug mode ON for execution of S.EXECUTE; same effect as entering S.EXECUTE in UTDB screen
-DH = Only valid on Distributed UniData deployments; Execute command on database server ('datahome')

<command> is a query language command such as SELECT

<arguments> are optional; is a string that contains arguments for the command (if applicable)

Example:
X.STMT = 'MIOSEL PERSON WITH LAST.NAME = 'Smith''
CALL S.EXECUTE (X.STMT)
FOR_EACH SELECTED ID
<code>
END_EACH SELECTED ID

The X.STMT line sets up the 'sentence' to execute. The next line that calls S.EXECUTE executes the statement. We now have an active list of PERSON records, of which we can use in the following FOR_EACH loop.

Field Process Order

UI 4 message box

CALL S_RIA_MESSAGES("", "Processing record: ": KEY.STUDENT.PROGRAMS)



S.MESSAGES options for the first parameter:

Here are the values for the first argument to S.MESSAGES: (copied from another posting)

1 = Use error processor; same effect as using other known methods of error handling in Envision processes

2 = Display message box; does not require user input to hide message; (Message will hide on its own at end of hook code)

3 = Display message; requires user interaction to hide message

4 = Same as option number 2

5 = Same as option number 3



* 1 Web version works completely, hangs in UI 2.3

* 2 UI version works completley, no message box displays on the Web

* 3 Web version works completely, hangs in UI 2.3





From <http://forums.datatel.com/viewtopic.php?f=28&t=35417&view=unread#unread>

Using S.POPMAIL to send emails in Colleague

XL.EMAIL.ADDRESS<1,1> = 'first@email.com'
XL.EMAIL.ADDRESS<1,2> = 'second@email.com'
X.EMAIL.SUBJECT = 'This is email subject'
X.FROM = 'sender@email.com'
XL.MESSAGE<1,1> = 'First line of body'
XL.MESSAGE<1,2> = 'second line of body
XL.MESSAGE<1,3> = 'third line of body'

XL.EMAIL.BODY = "From: ":X.FROM:@VM:@VM:XL.MESSAGE

CALL S.POPMAIL("S", XL.EMAIL.ADDRESS, X.EMAIL.SUBJECT, XL.EMAIL.BODY)

Envision Matrix and other things

I found a nice post on the forum from Patricia that could come in very handy for programmers.
Source: http://forums.datatel.com/viewtopic.php?f=28&t=12512


In the first question about adding the comments field to an existing LIST.VAR... You were only seeing the first line, because a list inside a list doesn't work. Further down in the thread, you saw where the CONVERT @VM TO \" \" IN ... converted the list for this line into a paragraph (sort of) by replacing the character that made it a list (@VM) with a space. Then, by putting the \"paragaph\" into the window's list variable, you would have been able to see all of it on a single row of the window (albeit, maybe limited by the size of the field, so maybe not see it all )

DIM = this is a straight UniBasic command to dimension a matrix in memory. In other languages, this may also be referred to as dimensioned arrays. Referenced in Envision and UniBasic with parenthesis, most other languages with square brackets. Since Envision is an extension of UniBasic, most of the UniBasic commands can be used in Envision. Matrices can be one dimensional or two dimensional. Think of a matrix similar to a spreadsheet.


Code:
DIM XM.TEMP(15) is a single row with 15 columns
DIM XM.TEMP.TWO(15,5) would be an entire page, 15 columns by 5 rows

MAT is the companion to DIM. Where DIM sets aside the size of the matrix, MAT is the command used to initialize the dimensioned array. You usually see it initializing a matrixto null

Code:
MAT XM.TEMP = \"\"
MAT XM.TEMP.TWO = \"\"
Matrices/dimensioned arrays can be very powerful and can be equally complex.

EQUATE is a command used by the compiler. It literally takes what you type and replaces it with the equated value in the object code. Usually equates are used for programmer readability. You see every table in Datatel has an assoicated \"equate table\" in the app.INSERTS (or maybe it's the app.SOURCE). For example


Code:
EQUATE LAST TO 1
EQUATE FIRST TO 3
Then when a record is read from the PERSON file, the individual fields are parsed into their v-dots using the equates.


Code:
V.LAST.NAME = R.PERSON<LAST>
V.FIRST.NAME = R.PERSON<FIRST>

is the same as saying V.LAST.NAME = R.PERSON<3>
An advantage of this approach is that something moves around to another loccation, you only need to change the equate statement and then recompile the code rather than change every place where you typed the literal location <3>.

PRINT versus X.LINE - PRINT is the UniBasic command to send stuff to the output device defined with the SETPTR command (like the printer or _HOLD_ file). PRINT_DETAIL is an Envision thing.

X.LINE is usefull for programmers to build their own line of data to be sent to the printer/output device rather than fighting with Envision's report writer... it's just easier some times

CRT is a UniBasic command to send stuff to the console/screen and is now banned in Envision.

SHOWA is the new GRSS debugger built into Envision by Datatel and would be the new replacement for any CRT statements you are using for debugging. It's really very cool. Look at the GRSS, GRS1 and GRS0 mnemonics and see if they have on-line help. It's the replacement for doing something like

Code:
IF DATATEL.DEBUG THEN
XL.DEBUG.MSG<1> = \"v.last.name =\":V.LAST.NAME
XL.DEBUG.MSG<1> = \"v.first.name =\":V.FIRST.NAME
CRT XL.DEBUG.MSG
END

would be replaced with the single line
SHOWA V.LAST.NAME; V.FIRST.NAME

Of course, to use SHOWA, you must use the Envision generator. If you are hacking generated source code (app.SOURCE), you would still use the UniBasic CRT command to get it to display to the console.

CALL versus CALL_SUBR - this is an Envision thing and has to do with the MIO buffers. CALL_SUBR is used by screens and enables DETAIL screens to cancel the commits to the database if it's parent is cancelled. CALL can be used in screens, but will write the data whether or not the calling screen is cancelled or not. CALL is also used in subroutines.