Filtering instances in CoCreate Modeling (08 Dec 2017)

The magic sauce: oli:sd-inq-obj-contents-sysid. The following code uses content sysids as the keys for a hash table. The unique-objects function returns the filtered list, i.e. a list which contains only one representative for any given number of shared instances.

(in-package :de.clausbrod.filterinstances)
(use-package :oli)

(defun unique-objects(objects)
  (let ((ht (make-hash-table :test 'equal)))
    (dolist (obj objects)
      (setf (gethash (oli:sd-inq-obj-contents-sysid obj) ht) obj))
    (loop for obj being the hash-values of ht collect obj)))

Expanding drawlists (14 Nov 2017)

The question centered on how to expand the results of the API call sd-inq-vp-drawlist-objects which returns a list of all currently visible objects.

In the example to the right, the following objects are checked, and are therefore visible:

• Assembly "a1"
• Part "cone" in assembly "a1"
• Part "cube" in assembly "a1"
• Part "backplate" in assembly "act_assy"
• Part "housing" in assembly "act_assy"
• Part "pistonhead" in assembly "act_assy/pst_subassy"
• Part "shaft" in assembly "act_assy/pst_subassy"

To reduce the amount of data it has to return, sd-inq-vp-drawlist-objects "compresses" its result as follows:

• If all objects below an assembly are checked (=visible), only the assembly is returned
• In partially visible assemblies, visible objects are returned individually

So in our example, sd-inq-vp-drawlist-objects would return a list containing:

• /a1
• /act_assy/backplate
• /act_assy/housing
• /act_assy/pst_subassy

This representation is perfectly fine for many purposes, but in the specific circumstances of the forum post, the user needed a way to "uncompress" the result, and wasn't interested in the assemblies, only in parts. So in the given example, the desired output would have been:

• /a1/cone
• /a1/cube
• /act_assy/backplate
• /act_assy/housing
• /act_assy/pst_subassy/piston-head
• /act_assy/pst_subassy/shaft

Assembly structures can be highly nested, of course, and so a recursive solution is needed.

My first solution revisited an approach I used in a previous blog post which discussed how to recurse over a hierarchy of objects and build a flat list from it.

(in-package :de.clausbrod.expanddrawlist)
(use-package :oli)

(defun flatten-assembly-mapcan(node)
  (cons node
   (mapcan #'flatten-assembly-mapcan (sd-inq-obj-children node))))

(defun expand-objects(objects)
  (loop for obj in objects
   nconc (remove-if-not #'sd-inq-part-p (flatten-assembly-mapcan obj))))

(defun show-part(p)
  (display (sd-inq-obj-pathname p)))

(let ((displayed-objects (sd-inq-vp-drawlist-objects (sd-inq-current-vp))))
  (mapc #'show-part (expand-objects displayed-objects)))

This worked and returns lists as described above. However, if you're actually not really interested in those intermediate lists, but instead simply want to visit all visible parts in the assembly tree and execute some action for each of those parts, then the problem can be solved more succinctly:

(in-package :de.clausbrod.expanddrawlist)
(use-package :oli)

(defun show-part(p)
  (display (sd-inq-obj-pathname p)))

(defun visit-parts(obj)
  (if (sd-inq-part-p obj)
      (show-part obj)
    (mapc #'visit-parts (sd-inq-obj-children obj))))

(let ((displayed-objects (sd-inq-vp-drawlist-objects (sd-inq-current-vp))))
  (mapc #'visit-parts displayed-objects)))

CoCreate Modeling: Wie startet man ein interaktives cmd.exe? (05 Jul 2016)

Es liegt nahe, das zunächst mit

  (oli:sd-sys-exec "cmd.exe")

zu versuchen. Das führt dann aber dazu, dass CoCreate Modeling scheinbar hängt und nichts Erkennbares passiert.

cmd.exe ist ein Kommandozeilenprogramm. Deswegen ist es völlig normal, dass ("grafisch") nichts passiert, wenn man cmd.exe als externes Programm ohne Parameter startet, zum Beispiel per sd-sys-exec. Dann wartet cmd.exe nämlich einfach im Hintergrund auf weitere Eingaben und tut sonst nichts.

Will man cmd.exe in einem eigenen Terminalfenster (landläufig "DOS-Fenster" oder "command shell" oder "command prompt") starten und interaktiv laufen lassen, kann man das so erreichen:

  (oli:sd-sys-exec "start cmd.exe")

(Zu den Kommandozeilenparametern und Besonderheiten des Helferleins start siehe http://ss64.com/nt/start.html.)

Bonusfrage: Wenn cmd.exe ein Kommandozeilenprogramm ohne grafische Oberfläche ist, wieso öffnet sich denn ein Terminalfenster, wenn man cmd.exe aus Windows Explorer heraus startet?

Antwort: Weil Explorer entsprechend vorkonfiguriert ist - intern wird in so einem Fall nicht einfach nur cmd.exe ausgeführt, sondern das moralische Äquivalent zu start cmd.exe.

Bonusfrage 2: Woher weiss Windows eigentlich, wo cmd.exe liegt? Muss man da nicht einen Pfad wie C:\Windows\System32\cmd.exe angeben?

Hintergrund: In der Forumsfrage wurde ein solcher hartkodierter Pfad verwendet.

Antwort: Das Verzeichnis, in dem cmd.exe liegt, taucht im Inhalt der Umgebungsvariablen PATH auf, die Windows beim Starten von Programmen konsultiert. Damit ist die explizite Angabe eines Pfades unnötig. Mehr noch, sie ist sogar kontraproduktiv und fehlerträchtig - denn nicht auf jedem Rechner liegt das Windows-Verzeichnis unter C:\Windows.

Bonusfrage 3: Wozu ist das eigentlich gut, so eine interaktive Instanz von cmd.exe aus einer CAD-Applikation heraus zu starten?

Kopfkratzende erste Antwort: Für sachdienliche Hinweise dankbar

Zweite Antwort nach Eintreffen der angeforderten sachdienlichen Hinweise: Ziel war es offenbar letztlich, ein kleines interaktives Kommandozeilenprogramm zu starten - der Start von cmd.exe war nur erster Test dafür.

A poor man's Common Lisp profiler (08 Mar 2016)

Alex's original code used quite some Lisp magic, including the little-known symbol-function which I elaborated about long time ago. But the code did not quite work yet. I gladly took the challenge. and ended up with a few lines of Lisp code which could profile (almost) any Lisp function in the system. The technique I used was to wrap the original function definition in a lambda closure. That closure is then installed using symbol-function.

(in-package :clausbrod.de)
(export '(profile-function unprofile-function list-profiling-results))

(let ((profile-hashtable (make-hash-table)))
  (defun profile-function(func)
    "Instrument function for profiling"

    ;; check if symbol-plist already contains profiler flag
    (unless (get func :profile-original-symbol-function)
      (let ((original-symbol-function (symbol-function func)))
   (when original-symbol-function
     (setf (get func :profile-original-symbol-function) original-symbol-function) ;; mark as profiled

     ;; install profiler code
     (setf (symbol-function func)
      (lambda(&rest r)
        (let ((start-time (f2::seconds-since-1970)))
          (unwind-protect
         (if r
             (apply original-symbol-function r)
           (funcall original-symbol-function))
            (let ((execution-time (- (f2::seconds-since-1970) start-time))
             (accum (gethash func profile-hashtable)))
         (if accum
             (setf (gethash func profile-hashtable) (+ accum execution-time))
           (setf (gethash func profile-hashtable) execution-time))
         (format *standard-output* "~%Execution time for ~S: ~,10F~%" func execution-time))))))
     ))))

  (defun unprofile-function(func)
    "Remove profiling instrumentation for function"
    (let ((original-symbol-function (get func :profile-original-symbol-function)))
      (when (remprop func :profile-original-symbol-function)
   (setf (symbol-function func) original-symbol-function))))

  (defun list-profiling-results()
    "List profiling results in order of decreasing accumulated execution times"
    (format *standard-output* "~%Accumulated execution times:~%")
    (let (table-as-list)
      (maphash (lambda(k v) (push (cons k v) table-as-list)) profile-hashtable)
      (dolist (pair (sort table-as-list #'> :key #'cdr))
   (format *standard-output* "~S:  ~,10F~%" (car pair) (cdr pair)))))
  )

(f2::win-open-console-window)
(setf si::*enter-break-handler* t)
(use-fast-links nil)

There are other profilers out there for Common Lisp, but it is not always straightforward to make them work in CoCreate Modeling which implements a subset of CLtL1 only. So who knows, maybe someone out there will actually find this useful!

To profile a function:

  (clausbrod.de:profile-function 'my-function)

Now execute my-function at your heart's content. Every time the function is called, the profiler measures its execution time. When the test session is completed, accumulated execution times can be listed as follows:

  (clausbrod.de:list-profiling-results)

And here is how to profile all functions in a given Lisp package:

  (do-external-symbols (s (find-package "FOO"))
    (when (function s)
      (clausbrod.de:profile-function s)))

My implementation differs almost entirely from Alex' version, which allows me to call it my own, but of course I owe thanks to Alex for starting the discussion in the forum and posting his original inspirational code!

The code is now available as a Github project, see https://github.com/clausb/lisp-profiler. There is even a simple GUI dialog on top of the low-level profiling code:

The version of the code shown above uses a SolidDesigner-specific way of getting the current time in high precision. The improved version in the Github project should work in other Lisp dialects as well. Fingers crossed.

CoCreate Modeling: Changing the current directory during startup (02 Nov 2015)

In the article, I showed how to subscribe to events in CoCreate Modeling (aka "PTC Creo Elements/Direct Modeling") for fun and profit. It turns out that this technique can also be applied to solve the following problem: In customization code which is loaded into CoCreate Modeling during startup, you want to set the user's default working directory to some default value specific to your environment or team.

You'd think that this should be trivial, as the current directory can be set using the IKIT's sd-set-current-working-directory API. But when you call this function during startup (i.e. from code in sd_customize, or in code loaded from there), you may find that other customization code or even CoCreate Modeling itself changes the current directory after your code runs. This is because CoCreate Modeling remembers the directory which was current before the user closed the last session. When you restart the application, it will try to "wake up" in precisely that working directory.

To override this behavior, here's a simple trick:

• In sd_customize (or, preferably, in code loaded from there), register an event handler for the SD-INTERACTIVE-EVENT.
• This event will be fired when startup has completed and the application becomes interactive.
• In the event handler:
  • Set the current working directory as you see fit
  • Unregister from the event (we want one-shot behavior here)

And here is what event handler code like this would look like:

(in-package :de.clausbrod)
(use-package :oli)

(defun interactive-event-handler(&rest r)
  (sd-set-current-working-directory (user-homedir-pathname))
  (sd-unsubscribe-event *SD-INTERACTIVE-EVENT* 'interactive-event-handler))

(sd-subscribe-event *SD-INTERACTIVE-EVENT* 'interactive-event-handler)

This particular event handler sets the current working directory to the user's home directory, but this is of course just an example for a reasonable default.

Flache Hierarchien (04 Apr 2015)

Die Aufgabe des Code-Stückchens war, in CoCreate Modeling eine Baugruppe abzuklappern und dabei zu verflachen. (Jaja, der aktuelle Produktname ist sowas wie PTC Creo Elements/Direct Modeling, aber spätestens beim Schrägstrich nicke ich weg.) Sprich: Für jedes Element in der Baugruppe wird ein Eintrag in der Resultatliste erzeugt, und diese Liste ist flach, also unverschachtelt.

In CoCreate Modeling werden Objekte repräsentiert durch sogenannte SEL_ITEMs - Lisp-Strukturen, die für Teile, Baugruppen, Arbeitsebenenen und allerlei andere Objekte in einem 3D-Modell stehen können. Damit man den Lisp-Code in diesem Artikel vielleicht auch einmal in einer anderen Lisp-Implementierung testen kann, definieren wir uns aber zunächst einmal eine extrem eingedampfte Sparversion als eigenen Datentyp node:

(defstruct node
  (name     ""  :type string)
  (children nil :type list))

Das reicht, um einen einfachen Teilebaum abzubilden. Ein Knoten kann entweder ein einfaches Teil repräsentieren - in diesem Fall hat er nur einen Namen. Wenn es sich um eine Baugruppe handelt, hält der Knoten eine Liste von Kindknoten in children.

(defmethod print-object ((node node) stream)
  (format stream "~A [~A]  "
     (node-name node)
     (if (node-children node) "asm" "part")))

Damit man einen node halbwegs kompakt ausgeben kann, definieren wir uns ein zur Struktur passendes generisches print-object. Aus der etwas langatmigen Darstellung einer Strukturinstanz wie

#S(NODE :NAME "42" :CHILDREN (#S(NODE :NAME "p42" :CHILDREN NIL)))

wird dadurch

42 [asm]  

Testbaugruppen baut man sich einfach per Strukturliteral. Beispiel:

(let ((tree #S(NODE :NAME "a1"
                :CHILDREN (#S(NODE :NAME "p1")
                           #S(NODE :NAME "p2")
                           #S(NODE :NAME "a11"
                               :CHILDREN (#S(NODE :NAME "p11")
                                          #S(NODE :NAME "p12")))
                           #S(NODE :NAME "a12"
                               :CHILDREN (#S(NODE :NAME "p13")
                                          #S(NODE :NAME "p14")))))))

Mit dieser Vorbereitung können wir nun endlich des Kollegen Codeschnippsel betrachten. Naja, eine leicht angepasste Variante davon jedenfalls:

(defun flatten-assembly-apply-nconc(node)
  (cons node
    (apply #'nconc (mapcar #'flatten-assembly-apply-nconc (node-children node)))))

Ruft man flatten-assembly-apply-nconc für die obige Testbaugruppe (flatten-assembly-apply-nconc tree), erhält man dank des von uns definierten print-object in der REPL in etwa folgendes:

(a1 [asm]  p1 [part]  p2 [part]  a11 [asm]  p11 [part]  p12 [part]  a12 [asm]  p13 [part]  p14 [part]) 

Es entsteht also in der Tat eine flache Liste - wie schön. Sich zu verbildlichen, warum die Funktion die gewünschten Effekt hat, braucht schon einen kleinen Moment - und vielleicht auch den einen oder anderen Blick ins Lisp-Manual, um sich der genauen Funktionsweise von nconc oder mapcar zu vergewissern. Entscheidend ist unter anderem, dass Lisp-Listen letztlich Ketten von cons-Zellen sind, deren letztes Element auf nil verweist, und dass node-children genau solche nil-Werte passend liefert, die von mapcar und nconc auch brav durchgeschleust werden.

flatten-assembly-apply-nconc setzt das "destruktive" nconc ein, um weniger Speicher allozieren zu müssen. Was mich gleich zu der Frage geführt hat, ob es vielleicht noch effizienter geht, und so entstanden folgende Varianten:

(defun flatten-assembly-apply-append(node)
  (cons node
    (apply #'append (mapcar #'flatten-assembly-apply-append (node-children node)))))

(defun flatten-assembly-mapcan(node)
  (cons node
    (mapcan #'flatten-assembly-mapcan (node-children node))))

;; version using an accumulator
(defun flatten-assembly-accumulator(node &optional acc)
  (cond
    ((null node) acc)
    ((listp node) (flatten-assembly-accumulator (first node) (flatten-assembly-accumulator (rest node) acc)))
    ((null (node-children node)) (cons node acc))
    ;; assembly case, i.e. a node with children
    (t (cons node (flatten-assembly-accumulator (node-children node) acc)))))

Diese Varianten habe ich hintereinander in drei Lisp-Implementierungen ausgemessen, und zwar in CLISP 2.49, Clozure CL 1.1 und SBCL 1.2.10. Weil SBCL sich zumindest auf Mac OS bei kurzläufigen Tests zickig anstellt und keine Messdaten liefert, habe ich die jeweilige Testfunktion in einer Schleife 100000mal aufgerufen:

(let ((tree #S(NODE :NAME "a1"
                :CHILDREN (#S(NODE :NAME "p1")
                           #S(NODE :NAME "p2")
                           #S(NODE :NAME "a11"
                               :CHILDREN (#S(NODE :NAME "p11")
                                          #S(NODE :NAME "p12")))
                           #S(NODE :NAME "a12"
                               :CHILDREN (#S(NODE :NAME "p13")
                                          #S(NODE :NAME "a121"
                                              :CHILDREN (#S(NODE :NAME "a1211"
                                                             :CHILDREN (#S(NODE :NAME "p1211")))))
                                          #S(NODE :NAME "p14")))))))

  (defun run-test(function-symbol)
       (gc)
       (format t "~%Test function: ~A~%" (symbol-name function-symbol))
       (print (time (dotimes (i 100000) (run-test-raw function-symbol)))))

  )

(run-test 'flatten-assembly-apply-append)
(run-test 'flatten-assembly-apply-nconc)
(run-test 'flatten-assembly-mapcan)
(run-test 'flatten-assembly-accumulator)

Die Angaben zu Zeit- und Speicherverbrauch lieferte dabei jeweils time.

Es gibt also durchaus signifikante Unterschiede im Speicherverbrauch. In CCL und SBCL liefert die Variante flatten-assembly-accumulator die beste Kombination aus Performance und Speichersparsamkeit. Für CLISP ist dagegen flatten-assembly-mapcan die vielversprechendste Alternative.

Weitere Vorschläge für Varianten? Bin gespannt!

PS: Natürlich ist das hier beschriebene Problem eine Variante der Aufgabe, eine verschachtelte Liste plattzuklopfen. http://rosettacode.org/wiki/Flatten_a_list#Common_Lisp hält einschlägige Lösungen hierfür parat.

PS/2: In der Lisp-Implementierung HCL, die in CoCreate Modeling verwendet wird, schneiden flatten-assembly-apply-nconc und flatten-assembly-mapcan am besten ab. Dies ist aber mit Vorbehalt gesagt, denn in HCL musste ich den Code - mangels Compiler-Lizenz - interpretiert ablaufen lassen, was das Performancebild vermutlich stark verfälscht.

And... Action! (Part 3, 19 Sep 2009)

Nothing of that kind. Instead, I put the blame on having too many users of a successful predecessor product

Let me explain.

In the 80s, our 2D CAD application ME10 (now: CoCreate Drafting) had become extremely popular in the mechanical engineering market. ME10's built-in macro language was a big success factor. Users and CAD administrators counted on it to configure their local installations, and partners wrote macro-based extensions to add new functionality - a software ecosystem evolved.

A typical macro-language command looked like this:

Users didn't have to type in the full command, actually. They could start by typing in LINE and hitting the ENTER key. The command would prompt for more input and provide hints in the UI on what to do next, such as selecting the kind of line to be drawn, or picking points in the 2D viewport (the drawing canvas). The example above also illustrates that commands such as LINE RECTANGLE could loop, i.e. you could create an arbitrary amount of rectangles; hence the need to explicitly END the command.

Essentially, each of the commands in ME10 was a domain-specific mini-language, interpreted by a simple state machine.

The original architects of SolidDesigner (now known as CoCreate Modeling) chose Lisp as the new extension and customization language, but they also wanted to help users with migration to the new product. Note, however, how decidedly un-Lispy ME10's macro language actually was:

1. In Lisp, there is no way to enter just the first few parts of a "command"; users always have to provide all parameters of a function.
2. Lisp functions don't prompt.
3. Note the uncanny lack of parentheses in the macro example above.

But then, we all know how malleable a language Lisp is. All of the problems above could be solved by a fairly simple extension with the following characteristics:

• Define a special class of function symbols which represent commands (example: extrude).
• Those special symbols are immediately evaluated anywhere they appear in the input, i.e. it doesn't matter whether they appear inside or outside of a form. This takes care of issue #3 above, as you no longer have to enclose extrude commands in parentheses.
• Evaluation for the special symbols means: Run the function code associated with the symbol. Just like in ME10, this function code (which we christened action routine) implements a state machine prompting for and processing user input. This addresses issues #1 and #2.

These days, you would probably use something like define-symbol-macro. Back then, the Common Lisp standard had not been finalized and our Lisp implementation did not provide define-symbol-macro yet. And thus, CoCreate Modeling's Lisp evaluator extensions were born.

To be continued...

And... Action! (Part 2, 08 Sep 2009)

No, we don't need contrived constructs like (print extrude) to show that extrude is somehow... different from all the other kids. All we need is a simple experiment.

First, enter extrude in CoCreate Modeling's user input line: The Extrude dialog unfolds in all its glory, and patiently awaits your input.

Now try the same with print: All you get is an uncooperative "Lisp error: The variable PRINT is unbound". How disappointing.

But then, the behavior for print is expected, considering the usual evaluation rules for Common Lisp, particularly for symbols. As a quick reminder:

• If the symbol refers to a variable, the value of the variable is returned.
• If the symbol refers to a function and occurs in the first position of a list, the function is executed.

extrude & friends belong to the symbol jet-set in CoCreate Modeling. For them, the usual evaluation rules for functions don't apply (pun intended). Using symbol properties as markers, they carry a backstage pass and can party anywhere. For members of the extrude posse, it doesn't really matter if you use them as an atom, in the first position of a list, or anywhere else: In all cases, the function which they refer to will be executed right away - by virtue of an extension to the evaluator which is unique to CoCreate Modeling's implementation of Common Lisp.

You can create such upper-class symbols yourself - using a macro called defaction. This macro is also unique to CoCreate Modeling. Functions defined by defaction are called, you guessed it, action routines.

But why, you ask, would I want such a feature, particularly if I know that it breaks with established conventions for Lisp evaluation?

Well, precisely because this feature breaks with the established rules.

To be continued...

"This software will crash Real Soon Now™. Promise!" (02 Sep 2009)

Yes, that's right; CoCreate Modeling was supposed to alert the user before an actual crash was about to happen - by applying rocket-science dynamic program analysis and prediction techniques, sacrificing chicken and roasting them on Intel CPUs, or by having programmers dance naked around bonfires of compiler manuals. Whatever it would take.

No doubt that such a feature would be highly valuable. Imagine working on a model for several hours, and then you drive the application into a crash, and both the application and your model data disappear forever. If you could predict the crash, you'd save everybody a whole lot of time and money. Oh, and with such code, you'd always win the lottery, too. How convenient

Fortunately, CoCreate Modeling has always had pretty elaborate crash handling mechanisms. Whenever an unexpected exception occurs, a top-level crash handler catches it, pops up a message describing the problem, causes the current operation to be undone, restores the 3D model to a (hopefully) consistent state, and returns the user to the interactive top-level loop so that s/he can save the model before restarting.

Over time, we taught our crash handler to deal with more and more critical situations. (Catching stack overflows and multithreading scenarios are particularly tricky.) Hence, users rarely lose data in CoCreate Modeling even if some piece of code crashes. Which pretty much obviates the need for the proposed clairvoyance module.

How to Detect Mergers & Acquisitions in Code (01 Sep 2009)

Let’s assume you’d do this by sending a URL to IE via DDE, and that you’d then verify the result by asking IE which page it actually loaded.

Let’s say that you’d use the URL of your company’s website, http://www.cocreate.com.

The day your QA people start yelling at you because the test fails miserably, you know that your company has been acquired, and that all accesses to http://www.cocreate.com have been automatically redirected to http://www.ptc.com

And... Action! (31 Aug 2009)

Reader highway45 recently came up with a very interesting observation (abridged and translated from German):

Usually, I call a dialog like this: (set_pers_context "Toolbox-Context" function)

Or like this: function

As soon as I add parentheses, however, the "ok action" will be called: (function)

When highway45 talks of "functions" here, he actually means commands like extrude or turn. So, (set_pers_context "Toolbox-Context" extrude)? Really? Wow!

set_pers_context is an internal CoCreate Modeling function dealing with how UI elements for a given command are displayed and where. I was floored - first, by the fact that an end user found a need to call an internal function like this, and second, because that magic incantation indeed works "as advertised" by highway45. For example, try entering the following in CoCreate Modeling's user input line:

(set_pers_context "Toolbox-Context" extrude)

Lo and behold, this will indeed open the Extrude dialog, and CoCreate Modeling now prompts for more input, such as extrusion distances or angles.

What's so surprising about this, you ask? If you've used CoCreate Modeling for a while, then you'll know that, as a rule of thumb, code enclosed in parentheses won't prompt for more input, but will instead expect additional parameters in the command line itself.

For example, if you run (extrude) (with parentheses!) from the user input line, Lisp will complain that the parameter "DISTANCE is not specified". But in highway45's example, there clearly was a closing parenthesis after extrude, and yet the Extrude command started to prompt!

So is set_pers_context some kind of magic potion? Try this:

  (print extrude)

The Extrude dialog opens and prompts for input! Seems like even print has magic powers, even though it's a plain ol' Common Lisp standard function!

Well, maybe there is something special about all built-in functions? Let's test this out and try a trivial function of our own:

  (defun foobar() 42)
  (foobar extrude)

Once more, the dialog opens and awaits user input!

So maybe it is neither of set_pers_context, print or foobar that is magic - but instead extrude. We'll tumble down that rabbit hole next time.

To be continued...

A package riddle, part IV (28 Aug 2009)

(defun test()
  (test_dialog))

(in-package :clausbrod.de)
(use-package :oli)

(sd-defdialog 'test_dialog
  :ok-action '(display "test_dialog"))

In part 3 of this mini-series, we figured out that the #: prefix indicates an uninterned symbol - and now we can solve the puzzle!

Earlier, I had indicated that sd-defdialog automatically exports dialog names into the default package. To perform this trick, somewhere in the bowels of the sd-defdialog macro, the following code is generated and executed:

(shadowing-import ',name :cl-user)  ;; import dialog name into cl-user package
(export ',name)                     ;; export dialog name in current package
(import ',name :oli)                ;; import dialog name into oli package
(export ',name :oli)                ;; export dialog name from the oli package

As a consequence, the dialog's name is now visible in three packages:

• The default package (cl-user)
• Our Lisp API package (oli)
• The package in which the dialog was defined (here: clausbrod.de)

This is quite convenient for CoCreate Modeling users - typically mechanical engineers, not Lisp programmers. They don't want to deal with the intricacies of Lisp's package handling, but instead simply assume that the command (dialog) will be at their disposal whenever they need it.

Let's look up what the Common Lisp standard has to say on shadowing-import:

shadowing-import inserts each of symbols into package as an internal symbol, regardless of whether another symbol of the same name is shadowed by this action. If a different symbol of the same name is already present in package, that symbol is first uninterned from package.

That's our answer! With this newly-acquired knowledge, let's go through our code example one more and final time:

(defun test()
  (test_dialog))

Upon loading this code, the Lisp reader will intern a symbol called test_dialog into the current (default) package. As test_dialog has not been defined yet, the symbol test_dialog does not have a value; it's just a placeholder for things to come.

(in-package :clausbrod.de)
(use-package :oli)

We're no longer in the default package, and can freely use oli:sd-defdialog without a package prefix.

(sd-defdialog 'test_dialog
  :ok-action '(display "test_dialog"))

sd-defdialog performs (shadowing-import 'test_dialog :cl-user), thereby shadowing (hiding) and uninterning the previously interned test_dialog symbol.

Until we re-evaluate the definition for (test), it will still refer to the old definition of the symbol test_dialog, which - by now - is a) still without a value and b) uninterned, i.e. homeless.

Lessons learned:

• Pay attention to the exact wording of Lisp error messages.
• The Common Lisp standard is your friend.
• Those Lisp package problems can be pesky critters.

The good news: If you follow a few rules of thumb, you'll probably never run into complex package problems like this. One such simple rule is to define your functions first before referring to them. So in our code example, defining the dialog first before loading/defining the (test) function would have saved us all that hassle.

Phew.

A package riddle, part III (22 Aug 2009)

(defun test()
  (test_dialog))

(in-package :clausbrod.de)
(use-package :oli)

(sd-defdialog 'test_dialog
  :ok-action '(display "test_dialog"))

Load the above code, run (test), and you'll get:

In CoCreate Modeling, the sd-defdialog macro automatically exports the name of the new dialog (in this case, test_dialog) into the default package. Hence, you'd expect that the function (test), which is in the default package, would be able to call that dialog!

Astute readers (and CoCreate Modeling's Lisp compiler) will rightfully scold me for using (in-package) in the midst of a file. However, the error doesn't go away if you split up the above code example into two files, the second of which then properly starts with (in-package). And in fact, the problem originally manifested itself in a multiple-file scenario. But to make it even easier for readers to run the test themselves, I just folded the two files into one.

Lisp actually provides us with a subtle hint which I ignored so far: Did you notice that the complaint is about a symbol #:TEST_DIALOG, and not simply TEST_DIALOG?

The #: prefix adds an important piece to the puzzle. Apparently, Lisp thinks that TEST_DIALOG is not a normal symbol, but a so-called uninterned symbol. Uninterned symbols are symbols which don't belong to any Lisp package - they are homeless. For details:

• Creating Symbols (from "Common Lisp the Language", 2nd edition)
• Programming in the Large: Packages and Symbols (from Peter Seibel's excellent "Practical Common Lisp")
• The Complete Idiot's Guide to Common Lisp Packages
• Potting Soil: Colons continued - uninterned symbols

Uninterned symbols are beasts which live in a slightly darker corner of Common Lisp, or at least you don't run into them too often. And in our particular case, it isn't exactly obvious how TEST_DIALOG turned into an uninterned symbol. We would have expected it to be a symbol interned in the clausbrod.de package, which is where the dialog is defined!

Those who are still with me in this series will probably know where this is heading. Anyway - next time, we'll finally solve the puzzle!

A package riddle, part II (20 Aug 2009)

To recap, here's the test code again:

(defun test()
  (test_dialog))

(in-package :clausbrod.de)
(use-package :oli)

(sd-defdialog 'test_dialog
  :ok-action '(display "test_dialog"))

Here is what happens if you save this code into a file, then load the file into CoCreate Modeling and call the (test) function:

"The function #:TEST_DIALOG is undefined"? Let's review the code so that you can understand why I found this behavior surprising.

First, you'll notice that the function test is defined in the default Lisp package. After its definition, we switch into a different package (clausbrod.de), in which we then define a CoCreate Modeling dialog called test_dialog.

The (test) function attempts to call that dialog. If you've had any exposure with other implementations of Lisp before, I'm sure you will say: "Well, of course the system will complain that TEST_DIALOG is undefined! After all, you define it in package clausbrod.de, but call it from the default package (where test is defined). This is trivial! Go read The Complete Idiot's Guide to Common Lisp Packages instead of wasting our time!"

To which I'd reply that sd-defdialog, for practical reasons I may go into in a future blog post, actually makes dialogs visible in CoCreate Modeling's default package. And since the function test is defined in the default package, it should therefore have access to a symbol called test_dialog, and there shouldn't be any error messages, right?

To be continued...

A package riddle (19 Aug 2009)

So here is the innocent-looking code:

(defun test()
  (test_dialog))

(in-package :clausbrod.de)
(use-package :oli)

(sd-defdialog 'test_dialog
  :ok-action '(display "test_dialog"))

Copy/paste this code into a file called test.lsp, then load the file into a fresh instance of CoCreate Modeling. Run the test function by entering (test) in the user input line. Can you guess what happens now? Can you explain it?

To be continued...

STEP files for the masses (29 Jul 2009)

The CoCreate Task Agent provides such functionality, but since it is an add-on module at extra cost, only some customers have it available to them. But that's no reason for despair, as it's pretty simple to add new functionality to the product.

Here's my take on the problem. My solution doesn't have any kind of glitzy UI, it doesn't handle errors, it's not optimized for performance - but it shows how the approach works, and that's all I wanted to accomplish.

;; (C) 2009 Claus Brod
;;
;; Demonstrates how to convert models into STEP format
;; in batch mode. Assumes that STEP module has been activated.

(in-package :clausbrod.de)
(use-package :oli)
(export 'pkg-to-step)

(defun convert-one-file(from to)
  (delete_3d :all_at_top)
  (load_package from)
  (step_export :select :all_at_top :filename to :overwrite)
  (undo))

(defun pkg-to-step(dir)
  "Exports all package files in a directory into STEP format"
  (dolist (file (directory (format nil "~A/*.pkg" dir)))
    (let ((filename (namestring file)))
      (convert-one-file filename (format nil "~A.stp" filename)))))

To use this code:

• Run CoCreate Modeling
• Activate the STEP module
• Load the Lisp file
• In the user input line, enter something like (clausbrod.de:pkg-to-step "c:/allmypackagefiles")

For each package (*.pkg) file in the specified directory, a STEP file will be generated in the same directory. The name of the STEP file is the original filename with .stp appended to it.

In pkg-to-step, the code iterates over the list of filenames returned from (directory). For each package file, convert-one-file is called, which performs the actual conversion:

For each of those steps, we use one of the built-in commands, i.e. delete_3d, load_package, step_export and undo. (These are the kind of commands which are captured in a recorder file when you run CoCreate Modeling's recorder utility.) Around those commands, we use some trivial Common Lisp glue code - essentially, dolist over the results of directory. And that's all, folks

Astute readers will wonder why I use undo after the load operation rather than delete_3d the model. undo is in fact more efficient in this kind of scenario, which is an interesting story in and of itself - and shall be told some other day.

What's in a name? (25 Jul 2009)

And by the way, John, thanks as well for all your work to keep the user group site going strong! When you, dear reader, stop by at the CoCreate user group forum next time, drop John a line to say "thank you". He deserves it.

I'm not done with all the renaming on the site yet; for example, there are a number of pages which I called "OsdmSomething" (silly me). "Osdm" is the abbreviation for "OneSpace Designer Modeling", which is yet another older name for what we now know as "CoCreate Modeling"...

As you can see, "CoCreate Modeling" was rechristened a lot in the past:

This is from memory - corrections most welcome. CoCreate users out there, what's your favorite product name?

I'm not a creative marketeer - just a lowly engineer and therefore not too imaginative. You see, I call my apples apples, my typos typos, and my bugs features. Simple and straightforward, that's what us engineers are like.

So personally, I would never have fiddled with the product name at all. Granted, some name changes were inevitable. After all, we separated from HP in 1996 and became independent as CoCreate, and so we couldn't use the "HP" prefix anymore, of course. And in late 2007, PTC acquired us, and our products needed to be integrated into PTC's portfolio. My own - way too simplistic - engineering approach to branding would have been:

• HP PE/SolidDesigner (colloquially: SolidDesigner)
• CoCreate SolidDesigner (colloquially: SolidDesigner)
• PTC CoCreate SolidDesigner (colloquially: SolidDesigner)

In fact, many of our customers still call the product SolidDesigner; apparently, that name stuck with people. And not quite incidentally, the name of the main executable on disk is SolidDesigner.exe

Anyway - I'll have to admit that I start to like "CoCreate Modeling" as well. It's reasonably short, simple to remember, alludes to what the product does, and it reminds users of our past as CoCreate - which is a nice nostalgic touch for old f*rts like me who've been with the team for almost two decades now...

I'd rather take it nice and slow - disabling 3D acceleration in CoCreate Modeling (15 Jul 2009)

First, check the latest driver versions provided by the graphics card vendor. If you already have the latest version, try a version which was previously certified either by PTC/CoCreate or one of the other large CAD vendors. Also, remember to switch the graphics driver into "CoCreate Modeling" mode if the driver has such an option.

If the problem persists, and the graphics card is supported by PTC, contact PTC for help. They will work with the graphics card vendor to fix the problem. If the card is unsupported, contact the graphics card vendor directly.

But if all this fails, or if you want to take a quick stab at the problem, you can also have CoCreate Modeling ask the graphics driver to turn off hardware acceleration for 3D graphics. This will often bypass the buggy parts in the graphics driver, and the problem will go away. Things will also slow down, of course, i.e. 3D viewing operations won't be as snappy as before. On most systems, however, you will still be able to work with small to medium assemblies just fine.

All you need to do to disable hardware acceleration in CoCreate Modeling is set an environment variable called SDPIXELFORMAT, and give it a value of SOFTWARE. To set the environment variable, use the System Control Panel.

Click sequence in Windows XP:

• Start/Control Panel
• Run System control panel
• Select the Advanced tab
• Click "Environment Variables".

Click sequence in Vista:

• Start/Control Panel
• Click System and Maintenance, then System
• Click Advanced System Settings; this may pop up a user-access control dialog which you need to confirm
• Click Environment Variables

Now create a new environment variable SDPIXELFORMAT and set the value to SOFTWARE.

Common Lisp in CoCreate Modeling (20 Jun 2009)

I loved the format because of the wide variety of topics presented. Also, this gave me the unique chance to present to this audience of hardcore Lisp geeks how we are using Common Lisp in our flagship 3D CAD product, CoCreate Modeling. Since I only had a few minutes, all I could do was skim over a few topics, but that's still better than a poke in the eye with C#

Not many in the audience had heard about our project yet, so there were quite a few questions after the presentation. Over all those years, we had lost touch with the Lisp community a bit - so reconnecting to the CL matrix felt just great.

Click on the image to view the presentation. The presentation mentions LOC (lines of code) data; those include test code.

Previous posts on the European Lisp Symposium:

• European Lisp Symposium
• Europan Lisp Symposium: Keynote
• European Lisp Symposium: Geeks Galore!
• Sightrunning in Milan

Aggressive predictions (03 May 2008)

I joined the company in 1991 when it was still called MDD, a division of Hewlett-Packard at their Böblingen site. Shortly after that, I heard somebody report that some sales guy from a competing company was questioning the viability of our division and our products. Apparently, customers were told something like: "Give them 2 more years, and then they're gone."

This was one of my first lessons how things are done in real bidniz: It's not just about the functionality and quality of your product, but also about how far customers will trust your company. And competitors will not hesitate to attack both on the product and the company credibility front.

Anyway - turns out that the prognosis was just a tad too aggressive. So far, we survived another seventeen years (and not just barely), way more than the two years the Cassandras gave us. And even as a part of PTC, we'll continue to serve our customers with our own product line, so the CoCreate story continues.

Memory of those days in the early 90s is hazy, and maybe I've got it all wrong. But then, I'm pretty sure somebody also mentioned which company that sales guy worked for.

It was PTC

So long, and thanks for all the functional fish! (30 Mar 2008)

The original implementation was in UCSD Pascal. A while ago, part-time Lisp hacker Frank Buß ported it to Lisp and added Postscript output, and he also posted a very nice description of his approach, illustrating how this example helped him understand how valuable support for higher-order functions in a language can be.

Frank's code is clear and compact, and the platform dependencies are all in one function, which made it easy to adapt to CoCreate Modeling's dialect of Common Lisp. In fact, all that's needed to run the code is the following loader code:

;; -*-Lisp-*-
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Description:  Wrapper to run Frank Buss' functional geometry code
;;               in CoCreate Modeling
;; Author:       Claus Brod  
;; Language:     Lisp
;;
;; (C) Copyright 2008 Claus Brod, all rights reserved
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;

(in-package :clausbrod.de)
(use-package :oli)
(export '(plot-escher))

;; Allow using lambda without quoting it via #' first
;; (No longer required in CoCreate Modeling 2008 and later.)
(defmacro lambda (&rest body)
  `(function (lambda ,@body)))

(defparameter *our-loadpath* *load-truename*)
(load (format nil "~A/functional.lsp"
              (directory-namestring *our-loadpath*)))

;; Modeling-specific plotter function
(defun plot-annotation (p)
  (let ((tempfile (format nil "~A/test.mac" (oli:sd-inq-temp-dir)))
        (scale 500.0))
    (startup::activate-annotation)

    (with-open-file (s tempfile
                       :direction :output :if-exists :supersede)
      (format s "line~%")

      (dolist (line (funcall p '(0 0) '(1 0) '(0 1)))
        (destructuring-bind ((x0 y0) (x1 y1)) line
          (format s "  ~D,~D ~D,~D~%"
                  (* scale (float x0))
                  (* scale (float y0))
                  (* scale (float x1))
                  (* scale (float y1)))))

      (format s "end"))

    (oli:sd-execute-annotator-command
     :cmd (format nil "input '~A'" tempfile))
    (docu::docu_vp :fit_vp)
    (delete-file tempfile)))

;; Shortcut for the Escher fish drawing
(defun plot-escher()
  (plot-annotation *fishes*))

The loader code adds the definition for the lambda macro which is missing so far in CoCreate Modeling, loads Frank's code, and then adds a plotter function which creates output in a 2D Annotation window.

Usage instructions:

• Download Frank's code from his site and save it as functional.lsp.
• Download the loader code and save it into the same directory.
• Load the loader Lisp code into CoCreate Modeling 2007 or higher.
• In the user input line, enter (clausbrod.de:plot-escher)

Thanks to Frank for this cute demo code!

Common Lisp at CoCreate (29 Dec 2007)

CoCreate OneSpace Modeling is a 3D CAD modeler built on a concept called "explicit modeling". C++ is used mainly in the modeling kernel, while Lisp dominates the UI, the application logic, many add-on modules, and the API. The Lisp part of the project is safely in the 7-digit lines of code range. I don't know much about the other large Lisp projects out there, and LOC isn't exactly the greatest way to compare application complexity anyway, so I'll play safe and just say that this is certainly a non-trivial Common Lisp project.

Back in 1995, when we were still a division of Hewlett-Packard, we published an article in "HP Journal" which outlines why we chose Common Lisp and what we're doing with it in the product. The article is still available at http://www.hpl.hp.com/hpjournal/95oct/oct95a7.pdf. Since then, the application changed a lot, of course - we even renamed it twice But much of what is said in the article about Lisp still applies.

The HP Journal article concluded:

Common Lisp is also used as a user accessible extension language for HP PE/SolidDesigner. It is a standardized, open programming language, not a proprietary one as in HP PE/ME10 and PE/ME30, and the developers of HP PE/SolidDesigner believe that this will prove to be an immense advantage.

SolidDesigner was the original product name; ME10 and ME30 were predecessor products which implemented their own little macro interpreters. Back then, we were a bit cautious about the potential benefits we'd reap, as the product was still in its early days. By now, however, we can say that Common Lisp was a key factor in helping a fairly small team of developers keep pace with the big guns in the industry, due to all the well-known productivity features in Lisp, such as macros, the REPL, or automatic memory management.

The HP Journal article describes how we use macros to define a domain-specific language called action routines, which are state machines which guide users through commands. Later, we extended that concept by automatically generating UI for those commands: Using the sd-defdialog macro, application developers can implement full-blown commands in just a few lines of code, without having to write any (or at least hardly any) code for services such as:

• Automatic "macro recording" of commands
• Context-sensitive online help
• UNDO support
• Command customization (commands can be started from toolbars, menus, a user input line, or from our "taskbar")
• Sequence control (dependencies of user inputs on other input)
• UI creation and layout
• Adherence to UI style guides
• Graphical feedback both in the UI and in 3D graphics windows
• Type and range checks for input data
• Automatic unit conversions (imperial to metric etc.)
• Prompting

I've been planning to blog more on sd-defdialog for some time, and hope to get around to it Real Soon Now.

Needless to mention, I guess, that I made sure that CoCreate is now also part of Peter's great list... .-)

PS: If you're interested, check out my other blog posts related to CoCreate Modeling or the CoCreate Modeling FAQ.

My job at CoCreate (01 Dec 2007)

I've been working with the fine folks of CoCreate for so long now that they tattoed an asset number on my forehead. In fact, I started at HP, in the very department (MDD, which stood for Mechanical Design Division) from which our CAD products were born, and later left HP, together with the rest of us CAD-heads .-), to form what is now known as CoCreate.

As a member of the OneSpace Modeling development team, I serve as the software architect for our OneSpace Modeling product line.

I specialize in all development areas related to general software engineering, as well as systems programming. That's why I deal a lot with things like operating system dependencies, memory management, file handling, globalization or 3D graphics. What I really like about my job at CoCreate is that I can still occasionally fiddle with the low-level, close-to-the-bare-metal stuff, and yet have a chance to work on higher levels of our software, too. Since OneSpace Modeling is a really big and impressive piece of software, I also got to learn an awful lot about managing large projects and about software development techniques. As a recent example, we have been introducing XP techniques into the lab over the past two or three years. Which is quite a ride, given that XP originated from much smaller projects. It's cool that even though we are an established company with a track record of close to 20 years or so, we're still flexible enough to integrate new approaches and learn from them.

My parents probably still don't have the faintest idea what my job is all about .-) But then, it is tricky to explain what it's like to work here without going into lots of boring details. If you really want to know, you could check out some of the articles which we wrote about SolidDesigner (as our product was called back then) in 1995 for HP Journal. In those days, I was working on the object management layer in our code: "Providing CAD Object Management Services through a Base Class Library" summarizes its purpose (local copy is here). If you think this is thrilling stuff, why don't you send us your résumé? .-) And if show me yours, I'll show you mine (access password-protected, contact me for details).

Sometimes, when I find a funky software development problem either at CoCreate or in my own projects, I blog about it in my software development blog, so if you like techno-babble, go check it out!

I also keep some job-related information in my XING and LinkedIn accounts.

What's my vector, Victor? (08 Aug 2007)

To make coordinate entry as simple as possible, the implementation of Lisp which is embedded in the product understands the following vector syntax:

  (line :two_points 100,100 0,0)

Common Lisp connoisseurs will notice that this is decidedly non-standard behavior. Those commas aren't supposed to be there; instead, commas serve their purpose in list quoting, particularly in macro definitions. (For a refresher, check out The Common Lisp Cookbook - Macros and Backquote.) And in any other implementation of Lisp, this code would indeed result in an error message such as "comma is illegal outside of backquote".

OneSpace Modeling's embedded Lisp, however, will notice a pair of literal numbers and assume that what the user really meant to specify is a structure of type gpnt2d, which holds x and y slots for the coordinates. And so what is really evaluated is more like this:

  (line :two_points (oli:make-gpnt2d :x 100 :y 100) (oli:make-gpnt2d :x 0 :y 0))

oli is the Lisp package which exports the gpnt2d structure as well as its accessor and constructor functions.

This explicit syntax is actually required whenever you need to specify coordinates using non-literals, such as when the actual coordinates are the results of mathematical calculations. For instance, vector syntax is not recognized in the following:

  (line :two_points (+ 50 50),100 0,0)

Now you'll get the expected error message reporting that "a comma has appeared out of a backquote". To make this work, you'd have to say:

  (line :two_points (oli:make-gpnt2d :x (+ 50 50) :y 100) 0,0)

But despite this limitation, the vector syntax extension was tremendously important for us: Coordinates can be entered in all kinds of places in the user interface where casual users would never suspect that what they are entering is actually thrown into funny engines which the propellerheads at CoCreate call "the Lisp reader" and "the Lisp evaluator".

I'm just a simple property list, I didn't expect the Spanish strinquisition! (19 Jul 2007)

"Now, you're the Lisp aficionado here, right", he said, "You've got to help me out: Strings don't work in property lists!"

Oh, great. Who knows, being regarded (undeservedly) as the local Lisp, ahem, expert may become a factor for my job security some day, so I thought I'd better keep a straight face. Besides, he was still standing close to that window, and I wanted to leave nothing but a reassuring impression on him.

On the other hand, what the heck was he talking about?

Frantically grepping my grey cells for information on property lists, I somehow recalled we sometimes use them as a poor man's hashtable, usually mapping keywords to flags. But it had been so long I used property lists myself that I even had to look up the syntax details. To avoid this embarrassment next time around, here are some notes.

A property list is associated with a symbol. This flat and unstructured list can be thought of as a sequence of indicator/value pairs, with the indicator being the "key", in hash map terms. So the list starts with an indicator, followed by a value, followed by an indicator and its value, and so on. This is how you usually set a symbol property:

  (setf (get 'some-symbol some-indicator) some-value)

And to inquire a symbol property, you just say something like (get 'some-symbol some-indicator).

some-indicator can basically be any type, and so I wasn't sure what my co-worker meant when he said that he couldn't get strings to work, until he explained the details to me: He was calling some Lisp-based API function in our product, and that function returns a property list. Unfortunately, that property list was special in that somebody had stuffed a string into it as an indicator, and so the property list looked somehow like this:

  ("foo" 42 "bar" 4711)

And indeed, if you now try to inquire the "foo" property using (get 'some-symbol "foo"), all you get is - nil.

To retrieve a property value, get walks the list and compares each indicator in the list with "foo" (in this example) - using eq. From which we can immediately conclude:

• The correct spelling of "property list" is p-e-r-f-o-r-m-a-n-c-e p-r-o-b-l-e-m, as each lookup requires traversing potentially all of the list.
• eq checks for object equality, not just value equality. Which means that things like literal (!) strings or characters cannot be indicators!

In our case, we say (get 'some-symbol "foo"), and that "foo" string literal creates a new string object. While that new object happens to have the same value as the "foo" string in the property list, it is not the same object.

Indeed, the Common Lisp HyperSpec is quite clear on that topic: "Numbers and characters are not recommended for use as indicators in portable code since get tests with eq rather than eql, and consequently the effect of using such indicators is implementation-dependent."

It all boils down to the simple fact that (eq "foo" "foo") returns nil.

Now hopefully we can fix the API which returned those inadequate property lists to my co-worker's code, but his code also needs to run in older and current installations, and so he needed a workaround of some sort.

His first idea was to get the property list and fix it up in a preprocessing step before using get or getf for lookup, i.e. something like this:

(defun fix-plist(plist old-indicator new-indicator)
  (let ((cnt 0))
    (mapcar 
      #'(lambda(item)
          (incf cnt)
          (if (and (oddp cnt) (equal item old-indicator))
              new-indicator item))
      plist)))

(setf my-symbol 42)
(setf (get 'my-symbol "indicator") "value") ;; mess up plist
(print (get 'my-symbol "indicator"))        ;; returns NIL
(print (getf (fix-plist (symbol-plist 'my-symbol) "indicator" :indicator) :indicator))

This works, kind of - but it is actually quite ugly. Sure, with this code, we should be able to safely move ahead, especially since I also closed that office window in the meantime, but still: I really hope I'm missing something here. Any other ideas out there?

"Macro" considered harmful (01 May 2007)

I used "macro" mostly for historical reasons. "Macro" is an overloaded term which can mean (too) many things:

• In Lisp, a macro is a piece of code defined by defmacro. Macros in Lisp are a clever way to extend the language. If you want to learn more about this (or about Common Lisp in general, in fact), I recommend Peter Seibel's "Practical Common Lisp" - here's the section on macros.
• CoCreate's 2D package, OneSpace Drafting, has a built-in macro interpreter which can be used to customize and extend the product. Since many OneSpace Drafting migrate from 2D to 3D, i.e. to OneSpace Modeling, they tend to take their nomenclature with them, and so they often call pieces of Lisp customization code a "macro", too.
• In many software packages, users can record the interaction with the product and save the result into files, which are then often called macro files. OneSpace Modeling's recorder is such a mechanism, and so using the word "macro" is kind of natural for many users.

And so many users of OneSpace Modeling call their Lisp functions and customizations "macros", although this isn't really the correct term. Well, at least in most cases. Many of those customizations use an API called sd-defdialog which is provided by the "Integration Kit" library which ships with OneSpace Modeling. This API is, in fact, implemented using defmacro, i.e. sd-defdialog is itself a Lisp macro. So if a user writes code which builds on sd-defdialog and then calls the result a macro, he's actually not that far from the truth - although, of course, still incorrect.

Die Kollegen Könige (21 Mar 2007)

Selbstverherrlichung, so lautete der Vorwurf, und Mißbrauch des Diskussionsforums als Werbeplattform. Ein Totreder sei ich zudem - und einer, der im Forum über andere CAD-Software meckere, "um damit von den eigenen Schwächen abzulenken."

Ich hielt es für unwahrscheinlich, daß ich diese Vorwürfe wirklich alle und vor allem in dieser Schärfe verdient hatte - aber solch starken Tobak ignoriert man besser auch nicht einfach so. Es traf sich, daß Urlaub angesagt war, und so ließ ich mir zwei Wochen Zeit, um das in Ruhe zu verdauen und zu begrübeln.

Danach nahm ich meinen zumindest zeitweisen Abschied aus dem Forum.

Was war geschehen, und warum dieser Schritt?

Anwender der Software, die wir bei CoCreate entwickeln und verkaufen, treffen sich in allerlei Foren, und in einigen davon bin ich Stammgast - besonders in den deutschen Foren. "Ich arbeite zwar bei CoCreate, aber ich schreibe das in meiner Freizeit und spreche nicht für die Firma" - das war als Signatur in jedem meiner Wortbeiträge zu lesen. Zwar hat CoCreate nichts gegen meine Beteiligung, aber ich war auch nicht im Auftrag der Firma zugange, sondern privat - ich war schlicht neugierig, wie Kunden mit der Software, die ich mitentwickele, umgehen und welche Erfahrungen sie damit machen.

Wie in allen Diskussionsforen, so gab es auch hier ab und an Reibereien. Selten jedoch verspürte ich so viel Gegenwind wie in den letzten Monaten, und zum ersten Mal in über sechs Jahren habe ich nun das Gefühl, daß schon meine bloße Anwesenheit zur Gereiztheit beiträgt.

Warum das? Nun, ich kann nicht in die Köpfe derer hineinsehen, die besonders genervt, zuweilen gar aggressiv auf mich reagiert haben. Aber zwei Spekulationen erlaube ich mir.

Anwender brauchen schnelle Lösungen, Entwickler gründliche

Wenn es im Produkt klemmt, will der Anwender möglichst fix eine Lösung, um weiterarbeiten zu können - selbst wenn die Lösung so hemdsärmlig und kurzlebig wäre, daß sie einem Softwareentwickler Magengrimmen verursacht.

Der Entwickler hingegen hat ein Interesse daran, Schwierigkeiten und Tathergang möglichst vollständig aufzuklären: Was ist der Kern des Problems, und welche Beobachtungen haben damit nichts zu tun? War es vielleicht doch ein Anwenderfehler und wie könnte man den in Zukunft vermeiden? Oder ist es ein Fehler in der Software, und wie kann ich den ohne Nebenwirkungen korrigieren, so daß ich mich später nie mehr darum kümmern muß?

Also fragt der Entwickler vier- oder fünfmal nach den genaueren Umständen, um die Lage zu sondieren und falsche Vermutungen auszuschließen. Naja, jedenfalls tue ich das gerne. Vielleicht habe mir auf diese Weise so nach und nach das "Totreder"-Image eingehandelt.

Kollege Kunde? Wohl doch eher König!

Auch wenn die Umgangsformen im Forum kollegial und locker sind, und auch wenn ich tausendmal betone, daß ich das Forum als Privatmann besuche: Aus Kundensicht stehe ich im Zweifel auf der anderen Seite und hafte für all die kleinen oder großen Probleme mit, die der Anwender mit CoCreate-Produkten oder mit CoCreate selbst hatte oder hat.

Im Forum geäußerter Werkstolz oder auch der Versuch, falschen Behauptungen entgegenzutreten, wird deswegen besonders kritisch beurteilt.

Simple Wahrheiten, denke ich heute - und daß die Vorstellung, mit Kunden feierabends am virtuellen Stammtisch klönen zu können, doch eher naïv war. Und wenn ich noch so darauf beharre, als Privatmann an den Diskussionen teilzunehmen: Das Verhätnis ist und bleibt nun einmal asymmetrisch.

Wäre die Betreuung von Foren offizieller Bestandteil meines Jobs, so müßte ich als beauftragter Vertreter meiner Firma mit Angriffen und Auseinandersetzungen leben - und könnte das dann auch gut, denn ich wäre ja nicht persönlich gemeint, oder zumindest könnte ich mir das plausibel einreden.

Ich war indes privat und aus Spaß an der Freud' dabei. Am Ende war vom Spaß wenig übrig, also hieß es für mich: Loslassen üben! Die deutschen Foren funktionieren schließlich auch ohne mich prima. Sehr wahrscheinlich besser als zuvor.

Bin ich eine Mimose? Gut möglich; ich weiß es nicht. Nur daß mir Auseinandersetzungen im Forum zuweilen die ganze Woche verdorben haben, das weiß ich. Und daß ich das nicht mehr erleben möchte.

Was bedeutet das nun für andere Foren? Dort läuft es besser. Vielleicht liegt es daran, daß dort in Englisch diskutiert wird und der Ton schon deswegen ein anderer ist. Jedenfalls werde ich einstweilen Foren wie das internationale CoCreate-Anwenderforum weiter besuchen.

Und dieser Blog? Und die FAQ-Seiten, die CoCreate-Produkte betreffen? Nun, auf dieser Website trifft sich offenbar ein anderes Publikum: CAD-Administratoren, Angehörige von Partnerfirmen, Programmierer. Diskussionen, die sich hier ergeben, haben in der Tat eher kollegialen Charakter. Ich mache hier also weiter.

Don't quote me on this (18 Mar 2006)

Let us assume that I'm a little backward and have a peculiar fondness for the DOS command shell. Let us further assume that I also like blank characters in pathnames. Let us conclude that therefore I'm hosed.

But maybe others out there are hosed, too. Blank characters in pathnames are not exactly my exclusive fetish; others have joined in as well (C:\Program Files, C:\Documents and Settings). And when using software, you might be running cmd.exe without even knowing it. Many applications can run external helper programs upon user request, be it through the UI or through the application's macro language.

The test environment is a directory c:\temp\foo bar which contains write.exe (copied from the Windows system directory) and two text files, one of them with a blank in its filename.

Now we open a DOS shell:

C:\>dir c:\temp\foo bar
 Volume in drive C is IBM_PRELOAD
 Volume Serial Number is C081-0CE2

 Directory of c:\temp

File Not Found

 Directory of C:\

File Not Found

C:\>dir "c:\temp\foo bar"
 Volume in drive C is IBM_PRELOAD
 Volume Serial Number is C081-0CE2

 Directory of c:\temp\foo bar

03/18/2006  03:08 PM    <DIR>          .
03/18/2006  03:08 PM    <DIR>          ..
01/24/2006  11:19 PM             1,516 foo bar.txt
01/24/2006  11:19 PM             1,516 foo.txt
03/17/2006  09:44 AM             5,632 write.exe
               3 File(s)          8,664 bytes
               2 Dir(s)  17,448,394,752 bytes free

Note that we had to quote the pathname to make the DIR command work. Nothing unusual here; quoting is a fact of life for anyone out there who ever used a DOS or UNIX shell.

Trying to start write.exe by entering c:\temp\foo bar\write.exe in the DOS shell fails; again, we need to quote:

C:\>"c:\temp\foo bar\write.exe"

And if we want to load foo bar.txt into the editor, we need to quote the filename as well:

C:\>"c:\temp\foo bar\write.exe" "c:\temp\foo bar\foo bar.txt"

Still no surprises here.

But let's suppose we want to run an arbitrary command from our application rather than from the command prompt. The C runtime library provides the system() function for this purpose. It is well-known that under the hood system actually runs cmd.exe to do its job.

#include <stdio.h>
#include <process.h>

int main(void)
{
  char *exe = "c:\\temp\\foo bar\\write.exe";
  char *path = "c:\\temp\\foo bar\\foo bar.txt";

  char cmdbuf[1024];
  _snprintf(cmdbuf, sizeof(cmdbuf), "\"%s\" \"%s\"", exe, path);

  int ret = system(cmdbuf);
  printf("system(\"%s\") returns %d\n", cmdbuf, ret);
  return 0;
}

When running this code, it reports that system() returned 0, and write.exe never starts, even though we quoted both the name of the executable and the text file name.

What's going on here? system() internally runs cmd.exe like this:

  cmd.exe /c "c:\temp\foo bar\write.exe" "c:\temp\foo bar\foo bar.txt"

Try entering the above in the command prompt: No editor to be seen anywhere! So when we run cmd.exe programmatically, apparently it parses its input differently than when we use it in an interactive fashion.

I remember this problem drove me the up the freakin' wall when I first encountered it roughly two years ago. With a lot of experimentation, I found the right magic incantation:

  _snprintf(cmdbuf, sizeof(cmdbuf), "\"\"%s\" \"%s\"\"", exe, path);
  // originally: _snprintf(cmdbuf, sizeof(cmdbuf), "\"%s\" \"%s\"", exe, path);

Note that I quoted the whole command string another time! Now the executable actually starts. Let's verify this in the command prompt window: Yes, something like cmd.exe /c ""c:\temp\foo bar\write.exe" "c:\temp\foo bar\foo bar.txt"" does what we want.

I was reminded of this weird behavior when John Scheffel, long-time user of our flagship product OneSpace Designer Modeling and maintainer of the international CoCreate user forum, reported funny quoting problems when trying to run executables from our app's built-in Lisp interpreter. John also found the solution and documented it in a Lisp version.

Our Lisp implementation provides a function called sd-sys-exec, and you need to invoke it thusly:

(setf exe "c:/temp/foo bar/write.exe")
(setf path "c:/temp/foo bar/foo bar.txt")
(oli:sd-sys-exec (format nil "\"\"~A\" \"~A\"\"" exe path))

Kudos to John for figuring out the Lisp solution. Let's try to decipher all those quotes and backslashes in the format statement.

Originally, I modified his solution slightly by using ~S instead of ~A in the format call and thereby saving one level of explicit quoting in the code:

  (format nil "\"~S ~S\"" exe path))

This is much easier on the eyes, yet I overlooked that the ~S format specifier not only produces enclosing quotes, but also escapes any backslash characters in the argument that it processes. So if path contains a backslash (not quite unlikely on a Windows machine), the backslash will be doubled. This works surprisingly well for some time, until you hit a UNC path which already starts with two backslashes. As an example, \\backslash\lashes\back turns into \\\\backslash\\lashes\\back, which no DOS shell will be able to grok anymore.

John spotted this issue as well. Maybe he should be writing these blog entries, don't you think?

From those Lisp subtleties back to the original problem: I never quite understood why the extra level of quoting is necessary for cmd.exe, but apparently, others have been in the same mess before. For example, check out this XEmacs code to see how complex correct quoting can be. See also an online version of the help pages for CMD.EXE for more information on the involved quoting heuristics applied by the shell.

PS: A very similar situation occurs in OneSpace Designer Drafting as well (which is our 2D CAD application). To start an executable write.exe in a directory c:\temp\foo bar and have it open the text file c:\temp\foo bar\foo bar.txt, you'll need macro code like this:

LET Cmd '"C:\temp\foo bar\write.exe"'
LET File '"C:\temp\foo bar\foo bar.txt"'
LET Fullcmd (Cmd + " " + File)
LET Fullcmd ('"' + Fullcmd + '"')  { This is the important line }
RUN Fullcmd

Same procedure as above: If both the executable's path and the path of the data file contain blank characters, the whole command string which is passed down to cmd.exe needs to be enclosed in an additional pair of quotes...

PS: See also http://blogs.msdn.com/b/twistylittlepassagesallalike/archive/2011/04/23/everyone-quotes-arguments-the-wrong-way.aspx and http://daviddeley.com/autohotkey/parameters/parameters.htm

-- ClausBrod - 27 Mar 2016

Two minute warning (16 Mar 2006)

Software is a freaky thing. I still don't know how to explain what it is to my mum and dad. I could tell them about ones and zeroes and how layer upon layer of hardware and software build on each other until they finally form what they see on a computer screen or in a digital camera. But I'm not overly confident I'd be able to bring across how this stuff really works, partially because I hardly think about the inner workings of computer systems anymore. The von Neumann architecture is so deeply engraved into Joe Developer's reasoning and mindset that it becomes an almost subconscious fundament of our daily work.

But many of those who use computers every day never really understand how this gadget works. They usually get by because over time the software industry has developed UI metaphors which shield users from the internal complexity. Until something unexpected happens - such as an application crash.

Once upon a time in a reality not too far away, a Japanese user started seeing application crashes. The bug report through which we learned about his problems did not really complain about individual bugs or situations in which the crashes occurred. Instead, he requested to add a feature to the software so that it would alert the user before a crash would occur so that he'd have the chance to save his data and exit before the crash actually happened.

Now this was not a request to add a top-level exception handler which kicks in when a crash occurs, reports the issue to the user and makes a last-ditch effort to save data in memory. We already had that in the application. No, what the customer really wanted our application to do was to predict that a crash was looming in the near future.

My brain starts to hurt whenever I think about this request. After all, a crash is usually caused by a hitherto undetected bug in the code, i.e. by an issue which neither we as the programmers nor the software itself know about. Being able to predict a crash which is due to a bug in our code is more or less equivalent to knowing that the bug exists, where it is located, what the user's next action will be, and whether that course of action would lead him into the "danger zone". I'll ignore the bit about predicting the user's action for a moment; but if either we or the software already knows about the bug, why not simply fix it in the first place rather than ceremonially announcing it to the user? (Did I miss something? Does any of the more recent CPUs have a clairvoyance opcode that we could use? )

It took me only a short while to explain this to our support folks, but then, they are sufficiently versed with software that they kind of "got it" naturally, even though most of them do not develop any software. I don't think, however, that we ever succeeded to communicate this properly to the customer.

Maybe I even understand the customer. He was probably thinking he was kind of generous to us; after all, he was willing to accept that any kind of software inevitably has some bugs, some of which even cause crashes, and that there is no practical way of dealing with this other than using the software and fixing the issues one by one.

But at the very minimum, he wanted to be warned. I mean, how hard can this be, after all! Even cars alert their drivers if there is a problem with the car which should be taken care of in a garage as soon as possible. Most of these problems, however, are not immediately fatal. The car continues to work for some time - you don't have to stop it right away and have it toed to the garage, but can drive it there yourself, which is certainly more convenient.

What seems to be a fairly simple idea to a customer, is a nerve-wrecking perspective for a developer. There is really no way to predict the future, not even in a computer program; this is what the halting problem teaches us. However, what seems obvious to a developer, sounds like a lame excuse to someone who is not that computer-savvy.

But then, maybe there are ways to monitor the health of software and the data which it processes, and maybe, based on a lot of heuristics, we could even translate those observations into warnings for users without causing too many false alarms...

Comment dit-on "knapsack" en français? (01 Mar 2006)

This week, a customer of our software asked a seemingly innocent question; given a set of tools of various lengths, he wanted to find subsets of those tools which, when combined, can be used to manufacture a screw of a given length.

From the description, I deduced that we were talking about a variation of the subset sum problem which is a special case of the knapsack problem. Faint memories of my time at university arose; I couldn't resist the weird intellectual tickle. Or maybe it was just the beginning of my pollen allergy for this year Anyway, I searched high and low on my quest to reacquire long-lost knowledge.

One of the weirder search results was a TV show called Des chiffres et des lettres which has been running for ages now on French TV. In that show, they play a game called "Le compte est bon" which is actually a variation of the subset sum problem! The candidates are supposed to solve this puzzle in about a minute or so during the show. Wow - these French guys must be math geniuses!

Anyway, I couldn't help but try a subset sum algorithm in Lisp. I ran it both using CLISP and the implementation of Lisp provided in CoCreate OneSpace Modeling. I started to collect some benchmark results for CLISP, comparing interpreted and compiled code to get a better feeling for the kind of improvements I can expect from the CLISP compiler. In the case of CLISP, the compiler improves runtime by roughly an order of magnitude. See the discussion of the algorithm for detailled results.

-- ClausBrod - 01 Sep 2017

Blame CoCreate, for instance (14 Feb 2006)

The company I work for is called CoCreate. The name was chosen because the company's mission is all about collaboratively creating things. That's all nice and dandy, but I guess the team who picked the name didn't include a programmer, and so they overlooked something pretty obvious which causes mild confusion every now and then.

Most programmers, when confronted with our company name, think of COM. After all, one of the most important functions in all of the COM libraries prominently displays our company name: CoCreateInstance. Now, if a programmer thinks about COM (and hence software) when she hears about us, that's probably fine, because, after all, we're in the business to make and sell software.

However, customers are not necessarily that technology-savvy, nor should they have to be.

A while ago, a customer complained that our software was sloppy because it wouldn't uninstall itself properly and leave behind traces in the system. Our installer/uninstaller tests didn't seem to confirm that. So we asked the customer why he thought we were messing with his system. "Well", he said, "even after I uninstall your stuff, I still get those CoCreate error messages."

The customer sent a screenshot - it showed a message box, displayed by an application which shall remain unnamed, saying that "CoCreateInstance failed" and mumbling some COM error codes!

It took us a while to explain to the customer that no, we did not install this CoCreateInstance thing on the system, and that it is a system function, and if we actually tried to uninstall it along with our application as he requested (kind of), he wouldn't be terribly happy with his system any longer, and that the other app was actually trying to report to the customer that it had found a problem with its COM registration, and that this should be looked after, not our uninstaller. Phew.

Now if only we had the time-warping powers of the publishers of "The Hitchhiker's Guide To The Galaxy", we'd send our company marketing materials back into time before Microsoft invented COM, and then sue the living daylights out of them. Well, if we were evil, that is

My memory took a little longer to swap back in, but while writing the above, it dawned on me that this incident wasn't the only one of its kind: Somebody had upgraded to a new PC and installed all applications except CoCreate's. Then, while syncing to his Palm Pilot, he got an "OLE CoCreateInstance Failed" error message, and started to search high and low on his shiny new PC for traces of CoCreate applications or components.

Puzzled, he posted to a newsgroup, and I replied with tongue-in-cheek:

Let me explain: When we kicked off CoCreate as a company, we sat together and thought about awareness strategies for the new company. So we called our buddies from Microsoft and asked them to name some API functions after us, and in exchange we would port our software to Windows NT. Neat scheme, and as you discovered on your system, the cooperation between the two companies worked just fine.

[... skipping explanation of the technical issue and hints on how to fix registry issue on the system ...]

The next step for CoCreate towards world domination will be to talk to some of our buddies in, say, Portugal, and offer them to develop a Portugese version of our application if they name their country after us.

Would I get away with a response like this if I was a support engineer? Maybe not. One more thing to like about being a software developer (Everybody in the newsgroup had a good chuckle back then.)

I'm so special (11 Feb 2006)

While learning Lisp, bindings and closures were particularly strange to me. It took me way too long until I finally grokked lexical and dynamic binding in Lisp. Or at least I think I get it now.

Let us consider the following C code:

  int fortytwo = 42;

  int shatter_illusions(void)
  {
    return fortytwo;
  }

  void quelle_surprise(void)
  {
    int fortytwo = 4711;
    printf("shatter_illusions returns %d\n", shatter_illusions());
  }

A seasoned C or C++ programmer will parse this code with his eyes shut and tell you immediately that quelle_surprise will print "42" because shatter_illusions() refers to the global definition of fortytwo.

Meanwhile, back in the parentheses jungle:

  (defvar fortytwo 42)

  (defun shatter-illusions()
    fortytwo)

  (defun quelle-surprise()
    (let ((fortytwo 4711))
      (format t "shatter-illusions returns ~A~%" (shatter-illusions))))

To a C++ programmer, this looks like a verbatim transformation of the code above into Lisp syntax, and he will therefore assume that the code will still answer "42". But it doesn't: quelle-surprise thinks the right answer is "4711"!

Subtleties aside, the value of Lisp variables with lexical binding is determined by the lexical structure of the code, i.e. how forms are nested in each other. Most of the time, let is used to establish a lexical binding for a variable.

Variables which are dynamically bound lead a more interesting life: Their value is also determined by how forms call each other at runtime. The defvar statement above both binds fortytwo to a value of 42 and declares the variable as dynamic or special, i.e. as a variable with dynamic binding. Even if code is executed which usually would bind the variable lexically, such as a let form, the variable will in fact retain its dynamic binding.

"Huh? What did you say?"

1. defvar declares fortytwo as dynamic and binds it to a value of 42.
2. The let statement in quelle-surprise binds fortytwo to a value of 4711, but does not change the type of binding! Hence, fortytwo still has dynamic binding which was previously established by defvar. This is true even though let usually always creates a lexical binding.
3. shatter-illusions, when called, inherits the dynamic bindings of the calling code; hence, fortytwo will still have a value of 4711!

Kyoto Common Lisp defines defvar as follows:

(defmacro defvar (var &optional (form nil form-sp) doc-string)
  `(progn (si:make-special ',var)
          ,(if (and doc-string *include-documentation*)
               `(si:putprop ',var ,doc-string 'variable-documentation))
          ,(if form-sp
               `(or (boundp ',var)
                    (setq ,var ,form)))
          ',var))

In the highlighted form, the variable name is declared as special, which is equivalent with dynamic binding in Lisp.

This effect is quite surprising for a C++ programmer. I work with both Lisp and C++, switching back and forth several times a day, so I try to minimize the number of surprises a much as I can. Hence, I usually stay away from special/dynamic Lisp variables, i.e. I tend to avoid defvar and friends and only use them where they are really required.

Unfortunately, defvar and defparameter are often recommended in Lisp tutorials to declare global variables. Even in these enlightened times, there's still an occasional need for a global variable, and if you follow the usual examples out there, you'll be tempted to quickly add a defvar to get the job done. Except that now you've got a dynamically bound variable without even really knowing it, and if you expected this variable to behave like a global variable in C++, you're in for a surprise:

  > (print fortytwo)
  42
  42
  > (quelle-surprise)
  shatter-illusions returns 4711
  NIL
  > (shatter-illusions)
  42
  > (print fortytwo)
  42
  42

So you call shatter-illusions once through quelle-surprise, and it tells you that the value of the variable fortytwo, which is supposedly global, is 4711. And then you call the same function again, only directly, and it will tell you that this time fortytwo is 42.

The above code violates a very useful convention in Lisp programming which suggests to mark global variables with asterisks (*fortytwo*). This, along with the guideline that global variables should only be modified using setq and setf rather than let, will avoid most puzzling situations like the above. Still, I have been confused by the dynamic "side-effect" of global variables declared by defvar often enough now that I made it a habit to question any defvar declarations I see in Lisp code.

More on avoiding global dynamic variables next time.

• Practical Common Lisp: Functions
• The Idiot's Guide to Special Variables and Lexical Closures

Fingerpointing at smart pointers (31 Dec 2005)

In an ATL COM client which uses #import to generate wrapper code for objects, I recently tracked down a subtle reference-counting issue down to this single line:

  IComponentArray *compArray = app->ILoadComponents();

This code calls a method ILoadComponents on an application object which returns an array of components. Innocent-looking as it is, this one-liner caused me quite a bit of grief. If you can already explain what the reference counting issue is, you shouldn't be wasting your time reading this blog. For the rest of us, I'll try to dissect the problem.

(And for those who don't want to rely on my explanation: After I had learnt enough about the problem so that I could intelligently feed Google with search terms, I discovered a Microsoft Knowledge Base article on this very topic. However, even after reading the article, some details were still unclear to me, especially since I don't live and breathe ATL all day.)

The #import statement automatically generates COM wrapper functions. For ILoadComponents, the wrapper looks like this:

inline IComponentArrayPtr IApplication::ILoadComponents () {
    struct IComponentArray * _result = 0;
    HRESULT _hr = raw_ILoadComponents(&_result);
    if (FAILED(_hr)) _com_issue_errorex(_hr, this, __uuidof(this));
    return IComponentArrayPtr(_result, false);
}

IComponentArrayPtr is a typedef-ed template instance of _com_ptr_t. The constructor used in the code snippet above will only call AddRef on the interface pointer if its second argument is true. In our case, however, the second arg is false, so AddRef will not be called. The IComponentArrayPtr destructor, however, always calls Release().

Feeling uneasy already? Yeah, me too. But let's follow the course of action a little bit longer. When returning from the wrapper function, the copy constructor of the class will be called, and intermediate IComponentArrayPtr objects will be created. As those intermediate objects are destroyed, Release() is called.

Now let us assume that the caller looks like above, i.e. we assign the return value of the wrapper function to a CComPtr<IComponentArray> type. The sequence of events is as follows:

• Wrapper function for ILoadComponents is called.
• Wrapper function calls into the COM server. The server returns an interface pointer for which AddRef() was called (at least) once inside the server. The reference count is 1.
• Wrapper function constructs an IComponentArrayPtr smart pointer object which simply copies the interface pointer value, but does not call AddRef(). The refcount is still 1.

Now we return from the wrapper function. In C++, temporary objects are destroyed at the end of the "full expression" which creates them. See also section 6.3.2 in Stroustrup's "Design and Evolution of C++". This means that the following assignment is safe:

  CComPtr<IComponentArray> components = app->ILoadComponents();

ILoadComponents returns an object of type IComponentArrayPtr. At this point, the reference count for the interface is 1 (see above). The The compiler casts IComponentArrayPtr to IComponentArray*, then calls the CComPtr assignment operator which copies the pointer and calls AddRef on it. The refcount is now 2. At the completion of the statement, the temporary IComponentArrayPtr is destroyed and calls Release on the interface. The refcount is 1. Just perfect.

Now back to the original client code:

  IComponentArray *compArray = app->ILoadComponents();

Here, we assign to a "raw" interface pointer, rather than to a CComPtr, When returning from the wrapper function, the refcount for the interface is 1. The compiler casts IComponentArrayPtr to IComponentArray* and directly assigns the pointer. At the end of the statement (i.e. the end of the "full expression"), the temporary IComponentArrayPtr is destroyed and calls Release, decrementing the refcount is 0. The object behind the interface pointer disappears, and subsequent method calls on compArray will fail miserably or crash!

So while ATL, in conjunction with the compiler's #import support, is doing its best to shield us from the perils of reference counting bugs, it won't help us if someone pulls the plug from the ATL force-field generator by incorrectly mixing smart and raw pointers.

This kind of reference counting bug would not have occurred if I had used raw interface pointers throughout; the mismatch in calls to AddRef and Release would be readily apparent in such code. However, those smart pointers are indeed really convenient in practice because they make C++ COM code so much simpler to read. However, they do not alleviate the programmer from learning about the intricacies of reference counting. You better learn your IUnknown before you do CComPtr.

This reminds me of Joel Spolsky's The Perils of JavaSchools, which is soooo 1990 (just like myself), but good fun to read.