Test in Filemaker

Introducing NORSULT_Functions, small tool to document and test all your Custom Functions in FileMaker Apps.

Read why I think you need to integrate a concept of software testing into the everyday development work.

As a starter, NORSULT provides the tool at the end of this article.

Integrate software test in FileMaker – is that necessary?

I admit, this is intentionally a provocative question. But hey, who integrates Test as a concept in her FileMaker programming?

We know since Dijkstra's notorious quote from the 1970ies that

„Program testing can be used to show the presence of bugs, but never show their absence!“

Dijkstra (1970) "Notes On Structured Programming" (EWD249), Section 3 ("On The Reliability of Mechanisms"), corollary at the end, p.7. (link)

What then is the goal with Testing software?

It would not be fair to leave that quote alone, Edgar W. Dijkstra, professor of mathematics in Eindhoven, NL, was uncompromising in his view that no complex automat (like computers, computer programs) should be just Trusted – they need to be engineered to be reliable. Part of this engineering process is to integrate the plans to test and verify the validity of the mechanism.

A software test thus must examine and validate the software for its obedience to defined rules and for its qualification to be applied to a given problem. In that sense testing measures quality of software. The minimum we can request from software is correctness. Tests should be designed to expose possible errors before the software is used in production. But there is no way to prove in general that any certain algorithm is error-free. But, as I once overheard, when it cannot be guaranteed that an algorithm returns a valid result in a timely manner (after a certain number of steps), it is practically useless, since a real-time understanding of correctness is violated.

Methodological approach

1. Be aware that the process of programming is error prone

2. Make sure that detectable errors get detected early in the process.

3. Integrate tests to assure the quality of your work.

4. Tests can be isolated and final only with very simple problems.

5. Use or create tools for testing.

How can this be translated to FileMaker programming?

1. FileMaker programming is error prone.

2. Very simple errors are avoided by the calc engine.

3. FileMaker has Test Helpers (Editor, Debugger, Data Viewer)

4. You must learn to integrate testing as a developer skill.

5. Isolated and final tests are almost impossible.

6. Test tools must be created.

Let us get more practical. To demonstrate my point, we do create a simple function.

fnAdder ( term1; term2 )

The goal is to produce a valid, correct result for any two distinct user inputs. The user inputs are assumed to be stored in two fields, FirstInput and SecondInput. The result shall be returned in an appropriate way…

Well, to start with, the description of the adder is not very exact. If the field FirstInput is defined to be a number field, then the input of ‚Hans‘ into that field would lead to the content representing 0 (numerical zero) when presented as number to the adder. Is that intended? If not, how do we avoid that input? Or even earlier in our thoughts, how do we learn to assume unintended input? And what is the appropriate way to return the result? Here is the key to the design of the program and the guide for designing its test.

Obviously, the only appropriate way to return the result of an addition is to return an integer. Or is it? It is up to you to decide when designing your program. Imagine a generic Adder, that can add text (i.e. concatenate) and add numbers…

All input that is intend must be planned for. Everything else must be set, tested and programmed in the pre- and post-processing conditions. If only integers qualify as input, only exact two integers must be provided before running the adder, and furthermore technical conditions have to be met (largest/smallest integer represented in the machine, result checking, mechanism for communicating detected failures…)

Try it out, and build a function in the data viewer that satisfies your requirements. And make that into a Custom Function. Document and run all test cases you can come up with. You probably end up with not being able to do this with only one custom function. And then react to a new requirement or a newly detected error condition. You will adapt the function definition and need to run all the tests again. Else you cannot guarantee that the last change did not break functionality already in place.

Getting back to methodology

What I suggested was to look at problemsolving in analytical way; dismantle complex problems into small chunks, built solutions for the chunks and test on this level (component test). Goal for testing is the proof of technical functionality and correct result. This can often be accomplished by the programmer herself. It is not the end of testing but a necessary start!

And I have one more thing.

The NORSULT f(x) tool

I used FileMaker to design a helper tool for our internal use. It should serve two goals: 1) easily document our library of Custom Functions and 2) have a fast and easy to use test tool. At the end of this blog entry is the link to download a community edition of the tool for free use.

What you can do with the tool.

First you have to import your existing library of Custom Functions into the file. Off course, you may start from scratch and define Custom Functions in this file. In any case, you can use the test tool built into NORSULT_Functions to build your documentation and test cases. How do you get the data into the file:

• Create a Database Design Report (XML Format) from a file that contains your custom functions.

• Use built in script to read the xml file (not the summary) and extract the definitions of not yet known custom functions (updates known ones.)

When you are done with the import, explore the the imported Custom functions.

On the left, just as in the picture above, all defined functions are listed. Clicking on one functions shows you the details on the right side of the screen. Look at the next picture for further explanation.

The name and expected parameters are automatically extracted, the body of the function definition is set to the corresponding field in the Custom Function Collection. The body of the function, containing the code, is extracted and presented in text form for documentation and easy access.

Dependencies are not yet auto detected, but can be entered. A missing function will then be detected. The use of a function is shown on the same page.

Since I did not build a parser for the text, you can highlight (select) the function name an drag it to the empty field in the 'used functions' section of the ‚Dependencies‘ Area:

If other functions use the current Custom Function, and they have been marked in the way described, they will show up in the right section ‚used by functions‘ in the ‚Dependencies‘ Area.

Now comes the real fun part. Click the button „Show Test Cases“ in the lower right corner.

A new window opens and you can create as many tests for the current function as you need.

Each time you want to run this test, press the button.

The test expression can be much more elaborated than in the example. You can write the expression in exact the same way as you would write in the expression window of the Data Viewer in FileMaker PRO Advanced tools.

When you are done with all your Custom Functions and have written test cases for them, there is a script to call which will run all tests in the order of their definition. Run the script and see that all test pass – or detect some error or hidden dependencies that break your test.

There is a demo version of the tool for you to download here: Download

For a demonstration and deeper discussion about how the tool and those CF-test relate to a larger picture of testing in FileMaker Solutions, come to see my session at FileMaker Konferenz in Hamburg, October 16.–19., 2019.